I remember a similar bug in the original Asherons Call (amazingly, still running with regular content updates!). Three or four years in, they added a new, very expensive object to the game (a "Pyreal Scarab" for anyone who cares). It cost a lot, so the idea was that people wouldn't be (at that point in the economy) buying many.
Now, in Asheron's Call, the world is huge. There are hundreds, if not thousands of vendors. And three of these vendors were set to sell their goods in stacks of up to 1000. Unfortunately, Cost of Pyreal Scrab * 1000 > 2^31, which wrapped. I can't remember if you either just got the goods for free (which you then sold back for huge profit), or if you actually got paid to take these things. Either way, overnight, the economy was destroyed. The entire game state had to be reset from backups; a dreaded rollback. Worse, the developer took a few days to do this.
Trust me, out of all the customers whose data you don't want to muss with, it's hardcore MMORPG players. Even though I was just a player, I can still remember the outrage all these years later. It taught me to always use appropriate types for objects with "value", and I've never accidentally used signed or floating point storage for currency again.
Haha - I was the one who was responsible for adding the Pyreal Scarab. I remember freaking out that night when I was playing and went to Teth's vendor and saw what was happening. I called up the producer at the time at 2am and said there was a "problem".
I remember this scenario distinctly because it happened within a month of me joining the game, and not understanding what was going on :) The bug was discovered (or at least widely known) about a day after the patch went live, and rolled back two days later. Nearly every bug that users felt were somewhat punishing led to players hitting the VNBoards asking about possible rollbacks so they could stop playing until it was done. I'm actually not sure if they've ever performed one since.
Consider this completely anecdotal, but I think that around the time this bug actually occurred, Asheron's Call and Everquest were the only two 3D MMOs that were worth mentioning. I recall the delay in rollback having something to do with Microsoft bureaucracy at the time as well -- Turbine was plagued with MS as a publisher having some sort of veto power over their business and was frequently met with resistance.
(Full disclosure: I love Turbine unconditionally for creating such memorable adventuring experiences with Asheron's Call 1 and 2)
That was the first thing which came to my mind when I read this, too. Of course, Asheron's Call didn't have an (official) real-money trading system to deal with too.
A nightmare scenario for the developers in both cases.
Asheron's Call had a very viable (unofficial) real-money trading system via ebay. I never even played the game but knew some people who did and they made fairly substantial amounts of money selling virtual crap on ebay. This was before the rise of the hardcore Chinese gold farming industry, in-game limitations on item trading (ala WOW), ebay restrictions, etc.
>Trust me, out of all the customers whose data you don't want to muss with, it's hardcore MMORPG players.
Welp, that's what happens when you have only online play on only official servers. Single player offline wouldn't be affected - cheat all you want! Online play on unofficial servers means server admins can take whatever action they want - ban offenders, leave offenders alone, or rollback - depending on what the admin and the players want.
Not any more than SimCity 2013 was an MMO. There may have been free-to-play MMO-like monitization of every little thing but the game mechanics are all single-player/small-group-of-players.
No, Asheron's Call was an MMO in every sense of the term. There was no free to play (at least in the pyreal scarab bug time period), monthly sub fee was it, and the game would have been wholly uninteresting without the full on massively multiplayer element.
Sorry, I was unclear. I was talking about Diablo III in my original comment and about it not having an offline mode or a mode where online play could be had on non-Blizard official servers (either dedicated server model or peer-to-peer).
This is what happens when we let machine constraints trump semantics.
When you say int, you usually want an actual integer, not an integer with an arbitrary limit. In this day and age, having that limit there is simply premature optimization.
I think having a nice bignum type--one that looks and feels just like a normal numeric type--is very important. It should also probably be the default; you should only switch to a machine type if you have a good reason. With gmp, big integers perform well enough to be used widely.
All Ruby code does this. Mathematics can still be performant because of tagged pointers. If the MSB is set, the value is a Ruby object. If it's not set, the value should be considered a native data type.
Perhaps I should have been more specific. I know that there are languages that have this property. My question is if there are studies on the performance impact. Preferably in a language that is suitable for high-performance applications (like games).
The python example is good here, because originally the Python language had separate types for unbounded-integer and machine-sized-int. This happened as part of the move from Python 2.x to Python 3.x, right along with changing strings from byte-arrays to unicode.[1]
There has been lots of fuss and even pushback about the change to use unicode instead of byte arrays, but I have never heard anyone complain about semantics OR performance of the switch to a single integer type.
I assume this was probably a server side bug, since all the accounting would never be trusted to the client side.
If you are writing highly-performant server code, the actually memory size is extremely important. You cannot (should not) abstract away the machine specifics of the datatype if you want to write optimized code.
In some cases where the underlying datatype isn't a concern (e.g. Javascript), I agree with you. But ultimately, this isn't a failure of technology, it is a failure of the software development process.
Most servers are x86-64 these days. So using 32 bit signed ints barely improves performance at all. Could have gone with a 64 bit unsigned int as it is a positive value all of the time.
A x64 CPU will take a single instruction to operate on a both a 64-bit and 32-bit integer, and the CPU registers are all 64-bit, so in that context you are right that it doesn't matter that much between the datatypes.
However, the physical size of the integer as stored in the CPU cache, RAM, and HDD is still going to be 2x as big for a 64-bit integer. In a hypothetical worst case, you are cutting your CPU cache and memory bandwidth in half, which is tragic.
Additionally, while most physical servers are x64, the OS, server software, and virtualization layer is still often 32-bit -- maybe for legacy, maybe for performance, maybe for a lot of reasons. Upgrading that whole stack up to 64-bits just for the luxury of having default 64-bit integers seems misguided.
Completely agree about big integers. I expect that the next generation of programming languages will offer a "number" type that's big integer or decimal by default, and then allow engineers to refine the type as an optimization. One trouble is the data storage layer - typical database fields are fixed size and don't naturally accommodate big integers. Thus where MMOs are concerned, using big integers will still require some deliberate effort.
Another approach is to have compile time checking of overflow or bounded numbers. There's been discussion about the utility of this on the Rust list recently and I wrote up some of my experience with languages that let you do checking: http://bluishcoder.co.nz/2013/05/07/ranged-integer-types-and...
Just another reminder that it's never worth bypassing the normal deployment process. Every year or two I learn a similar lesson myself: it's so tempting, the fix is so small, it couldn't possibly break anything (heck, I once had a data-gathering script that made a bunch of read-only calls to our system's API cause a live issue). Just say no.
The script was a permission checker; roughly, for each client check whether they had any object types they didn't have permission for, plus a bit more logic for things like "they have permission A which is a superset of B but don't have permission B", and output a report. There was a bug in the DB cache used by our security service (I think caused by a broken clone() implementation that shallow-copied when it should have deep-copied, but this was a while ago now) that meant that if permissions for two different clients were requested in quick succession, both would end up with the intersection of their permissions - which were then cached in the main application (I guess there must have been some logic like the API always triggered a refetch to ensure it returned the canonical value, whereas for the rest of the application performance was more important?). In theory it could've been happening already, but I guess we never had two clients loading the same webpage close enough to the same time (and they would have to both have had the local cache expire - plus almost all of our live clients had very similar permissions, whereas we had some "test" clients with no permissions at all), whereas of course my script just called as fast as possible.
It was fairly easy to track down the bug once we passed the initial disbelief stage, and I don't think many clients noticed. But it could've been a lot worse - I'd been so close to running this last-thing on a Friday before I left.
And I guess that meant that the clients would find themselves unable to access things since the permissions were cached, so you were actually seeing issues outside of your script results. Yea I can see how that would be truly baffling, and provides good motivation for not modifying interaction with production on a Friday evening. Thanks :)
Reminds me of a time a coworker at a previous job almost brought down a production server by opening a file in vi.
That file was a customer log that had grown to 100+gb in size due to an error that she was debugging. She failed to check the log size, instead assuming that it was a small file left over after that night's log rotation. When vi tried to load the file to memory, it almost crashed the box before we could kill it (we still got calls about degraded performance though).
I've made this mistake, only I was too naïve at the time to realise my mistake, thinking that vim, because of its reputation, could surely handle it. A server reboot later and I was enlightened.
Why wouldn't currency by handled by the type system? You could still have an overrun. But it'd be handled more appropriately.
Long ago, I wrote a budgeting / estimating tool. Costs were represented with binary coded decimals (BCDs). Not floating point numbers. Just like an accounting system.
Competing products could have weird roundoff errors. Not mine.
For accounting, why not just use scaled integers? Isn't there a limit to many many digits you want after the period? BTW, I think BCD is outmoded. Seems it's getting replace by DPD which has far less overhead (http://en.wikipedia.org/wiki/Binary-coded_decimal#Higher-den...).
To be fair, I am not sure if anyone would try posting 6 billion gold onto a PTR equivalent of the RMAH or whether the RMAH is even available on the PTR, which might explain why it wasn't tested.
Kingdom of Loathing had an integer overflow bug way back in 2004. It let players set their currency (in this case: meat) to the max value of a 64-bit integer. The game spent the next several months setting up meatsinks in an attempt to reduce the amount of meat in circulation.
Reminds be of a bug waaaay back. In the original Elite, you could obtained a missile lock on a space station, then dock and sell all your missiles, launch, and finally fire a missile (you still had the missile lock). Suddenly you have 255 missiles! 0-1 = 255 in unsigned 8 bit integer math!
You could then sell all your new shiny missiles for loads of money. Made a hard game a bit easier.
Those games (Frontier Elite and First Encounters included) had loaaads of bugs like that.
Another one in FE had you put in passenger holds, fill them with passengers, then sell the holds -- this would obviously not work as you had to evict the passengers first, however the game logic credited you with the cash anyway because the check came after the money had changed hands.
Yeah but that's true of most things that're not obfuscated. I used to use the Amiga Action Replay on my Amiga to alter memory on the fly in games like that.
Hard to understand why people spend so much time on a short game like Diablo III. Even before I finished it on inferno the game didn't make much sense, I just finished for the sake of it, which I regret.
Most Super Mario games requires way more abilities than that and less time.
Don't waste your limited time on earth playing consumption-driven games. I've been trying Eve online for a few days, It does not looks promising, it seems that Eve also is also driven by item accumulation and not actual playing.
It's basically a very pretty slot machine. I played quite a bit of D2 and D3. The thrill comes from grinding for hours (and hours and hours) and finding that one item that earns you hundreds of dollars. (Yes real dollars) You can thank the the Real-Money Auction House for both destroying the game and being the one thing that keeps so many people coming back for more.
I loved Diablo 2 and probably spent a good portion of my high school hours playing it. I was very excited for Diablo 3 and tried to get into it but just couldn't.
I felt that they just took a ton of ideas from WoW and implanted them into Diablo. Sure it may have made them more money but it made it a lot less fun.
Marvel Heroes is made by David Brevik - He had the original idea for diablo 1 and played a bigger role in diablo 2 than the guys that made torch light 2. Lets no mention his failed project with bill roper on hellgate london though.
I'm honestly surprised that they haven't been so classified yet. The theory-wonks had made this connection as far back as 2005, and the IRS wasn't entirely blind to it. (Though when Dibbell published Play Money in 2007, they had no clue how to deal with it.)
I guess with all the hysteria about video game violence, no one has had time to actually look at what's going on.
ETA: I'm guessing that the fact that there's some skill involved makes it more okay. Like a fly-fishing tournament.
Eve was a black whole for me. Played casually for a week, then learned how to make money via trading, then I found out there is a Python API for grabbing data out of the game client, as well as some other JavaScript-APIs using the in-game Browser.
Next thing you know I was crawling all popular market hubs in Eve, storing price history of each item in mysql, and programmatic analyzing the data to find the best trade routes for profit.
Then I realized I need much more data, and prepared a small data-grabber client for other people to run, as well some cloud storage to upload it to.
I looked at the calendar and noticed 2 weeks have past and I didn't do much else, so I came to the conclusion this might not be the most productive thing to do and quit Eve :) Problem is, I can't play these games the "normal" way, when I see it got APIs etc. I just have to go all out on it - or just not play at all.
Incidentally, Eve is the source of my favorite overflow bug! By putting a ship into a region of space that reduces the range of its weapons and using various range-debuffs, players were able to decrease the range of certain weapons by enough to get the range variable to roll over. This gave them near-infinite range on weapons that were intended to be short range. Since the short range was a tradeoff for very high damage, suddenly having infinite range was game breaking. However, the people who discovered the bug were clever, and did not abuse the power enough to be detected for nearly a year.
You're very wrong in that assumption. Don't get me wrong, I wasted 9 months of my life becoming completely absorbed by the game but it's so immersive that no other game can compare.
It's about hoarding, alliances, corporations, mining ops, PVP ops, big alliance battles, 0-sec space mining/pvp ops (this is the best part of the game).
I've heard Eve online called an animated space themed spreadsheet before. I tried it myself a long time ago, and I felt it was too complicated and boring.
Every time that I've been blown up in Eve, it's happened so quickly that the fight didn't even render on my screen. Just one second I'm in a spaceship, and the next I'm back at my home station. That feeling doesn't make me want to learn combat!
And you were probably travelling in a simple shuttle or cheap frigate, against multiple players or smartbombing idiots. Likely not consensual, and you were probably not expecting combat. Yeah, it sucks, but that too different from other MMOs (WOW, I am looking at you).
Many of my small-scale engagements took several minutes to resolve, one way or the other.
Not to mention fleets. In fact, one of my fondest memories was taking part in Agony Unleashed's PVP classes. That took several hours of intense playtime - we had to have multiple 'bio breaks' for people to be able to go to the bathroom and eat something - while being guarded and with scouts to give early warnings to the fleet. And it was amazing. Instructors teaching game concepts and military-like tactics and command and control (and actually enforcing discipline).
I've never seen a game getting even close to that sort of organization and discipline. CCP should hire those guys as teachers to introduce players to more advanced game concepts.
I played enough to get a lv. 60 Monk and lv. 60 Demon Hunter. The problem with Diablo III, compared to Diablo II from 10 years ago, is that nowadays there are so many options for online games without monthly fees that there's little incentive to stick to just one.
Then I started Guild Wars 2 a few months later, and played that to death. :)
Why? It's just a matter of scale. Until very recently, Romania's currency was such that most middle-class families had 8-10 digits in their bank account at any given moment, and the economy is relatively healthy.
Good items sell for hundreds of millions. The number of zeroes doesn't matter, as long as the balance between items and monetary value is stable.
"The number of zeroes doesn't matter, as long as the balance between items and monetary value is stable."
And you're not just starting the game. The way that D3 drops work, the first few times you're visiting the auction house, it's to buy, not sell (unless you get very lucky).
The game was also balanced around the auction house - meaning fewer high quality items drop, with the belief being that you're selling those few high quality items & kitting yourself out from the auction house.
When I first played, the difference between playing with just the drops I got versus kitting myself out from the auction house were vast. Some people might enjoy the challenge of just playing with dropped gear, but just as many do not.
" The game was also balanced around the auction house - meaning fewer high quality items drop, with the belief being that you're selling those few high quality items & kitting yourself out from the auction house."
AFAIK, Blizzard collects interest on each real money transaction, so if they're optimizing for maximization of that interest (I'm assuming they do) they aren't minimizing occurrence of the rarest of the items. They should behave like diamond cartel...
I think that's just it, the prices haven't remained stable, the economy just undergoes constant inflation as the amount of overall money keeps piling up. Meanwhile new characters still begin with nothing and earn gold at the same rate as players did when the game first launched.
Compare this with World of Warcraft where they deliberately built in mechanisms to keep destroying money through consumables and bind-on-equip items.
I feel sorry for the players who were amused by this and will now likely get banned. I don't know when video games started to be like a bad elementary school where you get punished for experimenting or finding a loophole, but it seems like it's punishing one of the fundamental joys of games. Or at least one of the joys I remember being particularly rewarding as a child.
I agree to some degree but MMO games in general are serious business and a special breed, especially when real currency is involved. It was most definitely against the ToS and I am sure that most of the players who participated in the exploit knew that. Also there were people who were using this bug to make real life money through the real money auction house, those people in particular can't be surprised that they were banned.
I think the difference is that this completely broke the economy, while the exploit in EVE was much more fixable. All they had to do in EVE was take away some people's loot and they were good.
Well, when you're messing with a real-money economy, I think that's a pretty big tip-off that you'll be punished for experimenting or finding a loophole. The same thing happens with other games that deal in real money, such as the stock market.
Also, AFAIK the only parties who exploited this bug were gold-farming bots. The computers won't mind, especially since their owners probably made bank off this.
It is odd that they're using 32-bit numbers when you'd be hard pressed to find a 32-bit only CPU and machines with over 4GB of memory are the standard.
I don't find it that odd. 32-bit applications are still very common, if not the standard. Most processes don't need more than 4GB.
More importantly, blindly increasing all of your 32-bit integers to 64-bit is going to double your memory usage, and ultimately just mask the real issue (i.e. improper bounds checking).
The actual money fields are probably small compared to the infrastructure used to track them. It's not going to double the amount of memory usage. I'd be surprised if memory use, network traffic and database I/O went up by more than a percent or two.
I agree for this one integer, they obviously should have been using 64-bit. I just take issue with the implication that Blizzard is somehow at fault or unusual for (presumably) not running a 64-bit stack.
They made a mistake for sure, but 32-bit vs. 64-bit architectures should not be on trial.
It's not about blindly increasing it, but using "long long" where normally "long" would be used, especially for values that have the possibility of wrapping.
The real impact on memory usage is likely the 8-byte pointers, but if you have a non-trivial amount of memory, it's rarely an issue worth fretting about.
If we cared about pointer sizes, we'd still be writing 16-bit code.
At the time the original code was written, it wasn't possible for a 32-bit integer to overflow, which is presumably why they didn't use 64-bit. That possibility was only introduced in the patch, and no one caught the bug that came with it.
Running 64-bit would have "prevented" this bug simply by virtual of that fact that the default datatype would have been big enough to avoid overflow, but it isn't really a solution. I just find 32-bit vs. 64-bit to be inconsequential to the real mistake, which was an improper software development process.
I mean in terms of bounded numbers as you might find in a game, or in real economies, not theoretically.
It's entirely conceivable that people have billions in currency, but not quintillions. Though perhaps Zimbabwe serves as a counter-point to this. They've had to slash twelve zeroes off their currency valuation on at least one occasion.
I remember World of Warcraft had a similar issue. The total amount of copper and in turn gold a player could have was the positive half of a signed 32-bit integer.
Very different issues. The total amount of the transaction is preserved. For this to be analogous you'd have to retain the copper after converting to gold and I'm not an expert on WoW but I don't ever recall that happening.
The scary thing is that integer overflows are considered rare so unlike things like null-pointer dereference no one really checks for them (heck, it seems impractical checking for it).
In this case, how should they defend against an overflow? Impose an arbitrary limit on gold?
I told this in their forums many times: you cannot have both real world money and one digital currency at the same time. Farming gold is hillarious. You need to implement instead something like bitcoin and it's the only way to stop the inflation.
Something like this used to work in Sim Farm too: buy and sell a piece of land and watch taxes grow until they flow over and you get a bunch of money instead of paying. If only this worked in real life :-)
Is this sort of hyperinflation intentional? Or a sign of economic ignorance? I don't get it. Gold is as common as dirt, and players are pumping huge amounts of it into the economy on a constant basis.
It's one of those things that makes English ... interesting to learn as a foreign language. :)
EDIT: Actually no, I mis-read the table on the Wikipedia page. The author was simply mistaken, and has corrected that. Thanks to everyone who pointed this out.
In Polish, there's one standard: million, milliard, billion. Every time I see some article in the press that tells about "billions" and seems to be based on some foreign data, I have to stop and think whether the author cited the original value unchanged, or converted it.
I wasn't aware of that. To be sincere, I also made the mistake of looking too close home. I was mostly talking about the non-English-speaking 'western hemisphere'.
Fixed. I suck at numbers. :P (or rather, fixed, but the server is getting hammered enough that it won't let me flush the cache)
There was another screenshot which has an account balance in the trillions that Blizzard argued was fabricated. Given the ease in which this trick was performed, I wouldn't be surprised if it was authentic.
Probably. I'm not a huge fan of Blizzard's business model and design choices as of late, but I do trust that they would not straight-faced lie versus fakeable screenshots. They know their back-end.
Heh, this reminds me of when I played Diablo 2. I duped so many sojs/items that I had multiple accounts just filled to the brim. I rarely play Blizzard games nowadays, but as long as there are games with any type of currency that can be sold, it will always be exploited.
Now, in Asheron's Call, the world is huge. There are hundreds, if not thousands of vendors. And three of these vendors were set to sell their goods in stacks of up to 1000. Unfortunately, Cost of Pyreal Scrab * 1000 > 2^31, which wrapped. I can't remember if you either just got the goods for free (which you then sold back for huge profit), or if you actually got paid to take these things. Either way, overnight, the economy was destroyed. The entire game state had to be reset from backups; a dreaded rollback. Worse, the developer took a few days to do this.
Trust me, out of all the customers whose data you don't want to muss with, it's hardcore MMORPG players. Even though I was just a player, I can still remember the outrage all these years later. It taught me to always use appropriate types for objects with "value", and I've never accidentally used signed or floating point storage for currency again.