When I started out I was basically stumbling around for code that worked. Things got a lot easier for me once I sat down and actually understood broadcasting.
The rules are: 1) scalars always broadcast, 2) if one vector has fewer dimensions, left pad it with 1s and 3) starting from the right, check dimension compatibility, where compatibility means the dimensions are equal or one of them is 1. Example: np.ones((2,3,1)) * np.ones((1,4)) = np.ones((2,3,4))
Once your dimensions are correct, it's a lot easier to reason your way through a problem, similar to how basic dimensional analysis in physics can verify your answer makes some sense.
(I would disable broadcasting if I could, since it has caused way too many silent bugs in my experience. JAX can, but I don't feel like learning another library to do this.)
Once I understood broadcasting, it was a lot easier to practice vectorizing basic algorithms.
This ramping up AI war will leave no prisoners. I am not an Apple customer in any way, I am in Google's ecosystem, but I feel that I need to make an exit, at least some essentials, preferably this year.
My e-mail, my documents, my photos, my browsing, my movement. The first step for me was setting up Syncthing and it was much smoother than I initially thought. Many steps to go.
I’m writing this comment so that people who want to know more about alternative theories of consciousness (to materialism/physicalism [1]) can know where to go to find well-argued positions on the topic.
(To be clear, I’m not here to argue about the topic or try to persuade anyone of any position – that’s a waste of everyone’s time).
I recommend seeking out discussions involving:
- Federico Faggin: inventor of silicon-gate technology and developer of the earliest microprocessors;
- Bernardo Kastrup: Ph.D. in computer engineering (reconfigurable computing, artificial intelligence), former CERN engineer at the LHC;
- Donald D. Hoffman: Ph.D. in computational psychology, professor in Cognitive Sciences at UC Irvine.
On YouTube you can find plenty of discussions involving these figures, some with each other, and plenty more with others.
I’d suggest it’s particularly important to explore these discussions as dispassionately as possible if you regard materialism as the only theory of mind that has any scientific credibility or validity.
As Christopher Hitchens reminds us in his legendary oration on John Stuart Mill and free speech [2], it’s only by thoroughly understanding the opposing view that we can thoroughly understand our own position on any topic.
My undergrad math professor created one of the first fully online linear algebra texts: http://linear.ups.edu/html/fcla.html It's integrated with Sage, a Python library for studying (among other things) number theory. Another prof at the same university also wrote his own linear book, using a lot more illustrations, but as a traditional textbook.
I see this book as a solid evolution in both directions. Nicely done!
This is my semi annual plug for all of you to watch the fantastic and somehow forgotten FX network TV show, The Americans, A spy drama set in Washington DC in the 1980s about KGB “illegals” posing as travel agents.
It’s way better than any basic cable TV show had any right to be. Plus, all seasons are streaming on Hulu, so you don’t have to worry about whether the story will be completed.
The most advanced math I had were the 1st and 2nd year multivariable calculus and linear algebra courses in college.
I am interested in visualizations/simulations of physical systems as a way to learn advanced math. Is there any books or resources that take that approach?
I don't know whether the article is especially relevant, but sea surface temperatures have gone absolutely nuts for the past year and so anything we would have called a crazy model five years ago is now on the table.
We spent most of 2023 (April to December) 4 to 5 standard deviations above the mean for 1981-2011. This year has been higher than Q1 2023's figures by a considerable margin.
This might have something to do with decreasing sulphur dioxide concentrations from switching shipping over to cleaner fuel, or it might not.
The most recent(?) state plan: https://ww2.arb.ca.gov/sites/default/files/2022-11/2022-sp.p... . On page 203 you see the plan is 20GW of offshore wind, an enormous amount of storage and new solar, and smaller contributions from everything else. I'd bet enhanced geothermal and heat and iron storage would be weighted higher today. If we can permit it, we'll do it.
London, where the main problem is (two people on minimum wage can buy a 3 bed house pretty much everywhere in the country outside the south east) has under 2.5% either empty, in use for short term lets like airbnb, or used for people who have a flat in London and a family home elsewhere.
There were some developments using LLMs in the timeseries domain which caught my attention.
I toyed with the Chronos forecasting toolkit [1], and the results were predictably off by wild margins [2]
What really caught my eye though was the "feel" of the predicted timeseries -- this is the first time I've seen synthetic timeseries that look like the real thing. Stock charts have a certain quality to them, once you've been looking at them long enough, you can tell more often than not whether some unlabeled data is a stock price timeseries or not. It seems the chronos LLM was able to pick up on that "nature" of the price movement, and replicate it in its forecasts. Impressive!
I think some of the financial applications around LLMs right now are better suited for things like summarization, aggregation, etc.
We at Tradytics recently built two tools on top of LLMs and they've been super popular with our usercase.
Earnings transcript summary: Users want a simple and easy to understand summary of what happened in an earnings call and report. LLMs are a nice fit for that - https://tradytics.com/earnings
News aggregation & summarization: Given how many articles get written everyday in financial markets, there is need for a better ingestion pipelines. Users want to understand what's going on but don't want to spend several hours reading through news - https://tradytics.com/news
The biggest current limitation with cloud providers when it comes to exchange tech is the lack of real multicast support. It is rare outside of exchanges, but extremely low latency L1 multicast market data has become the backbone of exchanges, both for fairness and for scalability.
Knowing you can saturate your entire network with 10G traffic and every participant will get the same market data packets at the same time[0], and there will be zero queuing or bottlenecks is very hard to do otherwise. There is a pretty good podcast episode about it out of Jane Street[1].
I know AWS have 'multicast support' but last time I tested it, it was clearly just uni-cast traffic with a software switch doing fan-out/copying, I assume using the same tech as their transit gateway, I think it was called hyperplane or something.
[0]: for some definition of the same time, at least low enough that you can't measure it without equidistant optical splitters or White Rabbit synced devices.
I enjoyed MIT's Fundamentals of Manufacturing course, on edx [0]. The team talk about casting, injection molding, 3D printing, and a couple of other ways.
Walk every aisle of your local Best Buy or MicroCenter. Do a mental calculation of how would this hardware product be 10x better with (present day’s) latest software trend.
Eg can I make this blender 10x better by bolting it to an LLM? Can I make this coffee machine 10x better by adding on a coffee+distilled water+third wave salt subscription? Can I make this network router 10x better shipping it with an invisible-to-normies tailscale + Speedify + 5G SIM card for ultra low latency, highly reliable zoom calls from the middle of nowhere using fast open and MPTCP?
Why 10x? Because if it’s not a 10x better experience the switching costs (aka “activation energy”) will be too high to gain market traction before you run out of funding and die.
Stack rank your items by categories such as “will this increase usage from once every 9 months to 9 times a day (ie smoke detector to co2 environmental sensor)”. “Does this have have a viral k factor that the original didn’t (access control for package delivery in door locks)”. “Does this unlock a recurring revenue stream that previously wasn’t there?” (smart decoding of video feed events). Etc.
Narrow your list down from 220 to 5. Build some prototypes. Test your market amongst fellow founders. Find the one niche vertical that has surprising utility and becomes a runaway success which exceeds your ability to fabricate in your living room using arduinos and off the shelf parts until you’ve exhausted Best Buy, Amazon, Spark Fun, and Adafruit’s supply of said item and you need to make your own version.
Get partnered up with a second rate design firm in one of the flyover states that has weird connections for manufacturing in this space from a random guy you met a JS conference three years earlier that cold calls you. Launch a successful crowd funding campaign. Raise funding. Fly to China to meet some contract manufacturers.
Pick the wrong contract manufacturer based on your own inexperience and your design firm’s prior relationship. Have things go terribly wrong because one didn’t use glass reinforced abs instead of regular abs because injection molded tools are expensive and a lot of tooling adjustments boil down to “guess and check”.
Move to China. But like not the cool part like Shenzhen or Shanghai - like a semi-obscure part - like Yiwu. Have way too many hot pots over Mao Tai trying to find a T2 vendor that knows how to injection mold helical gears. Learn some gutter slang from an obscure dialect of Chinese like Gan. Fire half a dozen T2’s. Hire five more. Fire three more. Refine it down to one final T2 that’s giving you problems. Eg try to find a paint vendor that can match the white on your injection molded part to the white on your metal part.
Nope out of the first paint vendor who can actually do this because of sketchy work conditions like a half naked infant running around the factory floor wearing 开裆裤.
Almost rage quit.
Question why you’re pirating grad school text books on material science from Turkish warez sites instead of taking that aqui-hire to AirBnB when they were like 12 people. Make a futile effort to give out safety equipment to T2’s regarding hearing and eye protection because you begin to question how everything is done in this country.
Nearly burn out again.
Begin to see the light. Parts fit. Production is scaling. Find a critical flaw. Fail.
Then do it all again until you succeed or run out of money.
In my experience, Supermaven makes Copilot look like a joke, and they’ve just released a Jetbrains plugin. YMMV. It’s just code suggestions though, no chat box.
Mathematically, the Fourier transform is "simply" a way of representing time signals in a certain orthogonal vectorial basis. Vectors in an ordinary sense, e.g. a displacement vector on Earth's surface can also be represented in several orthogonal bases: one basis could, for example, be two vectors pointing North and East; another could be a vector pointing along a certain road and one perpendicular to it. There is nothing inherently special about any of these bases, one could draw maps according to any of these two or many other conventions. (Orthogonal basis vectors are not even necessary, only convenient.)
The interesting thing about time-dependent signals (or any "pretty" function, really) is that they live in an infinite-dimensional vector space, which is hard to imagine; but (besides some important technicalities) the math works mostly the same way: signals as infinite-dimensional vectors can be represented in a lot of bases. One representation is the Fourier transform, where the basis vectors are harmonic functions. The "map" showing the shape of a signal as a combination of infinitely many harmonic functions -- i.e. the frequency domain -- is just as real as any other map with different basis vectors, e.g. the Walsh–Hadamard transform mentioned in the article. And, crucially, the original time-domain representation is also just one map showing us the signal, though it is often the most natural to us.
I have extensively tested Optuna and Weights and Biases (WandB) for hyperparameter tuning on multiple task-specific transformer models back in 2020.
Optuna lost out by a long mile back then in feature parity and dashboarding. Optuna did not have hyperband optimization which was and still is one of the best search algos for hyperopt. It looks like it is possible to implement hyperband yourself now, but in the loosely coupled architecture between Sampler and Pruner it's a bit baroque [1].
Anyway back then it was clear WandB was the far superior choice for features, ease of use, experiment tracking and dashboarding. We went with WandB for our lab.
Could be Optuna caught up, but WandB has seen significant development too. Looking at their dashboard docs, it looks meagre compared to what you can do with WandB.
Still rocking my EPYC 7282 in my home server, which really sits in a sweet spots: 16 Cores, about $700, 120W TDP (because of the reduced memory bandwidth).
Looks like the 7303 fills that same niche in the Milan generation (and should be compatible with any ROME mainboard, possibly after a BIOS update), or if you're building a new system you can get the 32-Core Siena 8324PN for about 130W TDP.
(While it may be silly to look at TDP for a server CPU, it does matter for home servers if you want a regular PC Chassis and not a 1U/2U case with a 12W Delta cooling fan that is audible three cities over. In fact, you can get the 8-Core 80W 8324PN and still get all those nice PCIe lanes to connect NVMe SSDs to, and of course ECC RAM that doesn't require hunting down a specific motherboard and hoping for the best.)
