You are really twisting things to make your argument sound plausible in the general case. More supply means less wages. Why focus on low paying jobs? Are you seriously suggesting that if we import every software engineer from India that wants to come here that my salary will increase? If so, that's very interesting why tech CEOs are lobbying so hard for this.
Your "more supply equals lower wages" argument is demolished by top economic research. A recent NBER study calculated that "immigration, thanks to native-immigrant complementarity and college skill content of immigrants, had a positive and significant effect between +1.7 to +2.6% on wages of less educated native workers" between 2000-2019.
The economy isn't zero-sum. As Milton Friedman noted, "most economic fallacies derive from the tendency to assume that there is a fixed pie, that one party can gain only at the expense of another." Immigrants create demand for housing, food, education, entertainment, and specialized services that natives often provide.
Historical evidence consistently disproves the fallacy: When women entered the workforce, it didn't cause massive job losses among men. When segregation was abolished, Black workers didn't cause mass unemployment among whites. The vast majority of Americans descend from immigrants who contributed to economic growth.
Research on H-1B visas shows that firms that get immigrant labor end up "hiring more tech workers and paying them more, because they become more efficient and sometimes scale up." In fact, studies show each H-1B worker creates approximately 1.83 jobs for native-born Americans.
The UK's Migration Advisory Committee, after reviewing studies from 2003-2018, concluded that "immigration had little or no impact on average employment or unemployment of existing workers" and "little impact on average wages."
The overwhelming consensus among economists is that immigration grows the economic pie rather than merely redistributing slices. That's why America's most immigrant-rich cities consistently have the highest wages, not the lowest.
PLEASE, I am begging you. Spend 15 minutes reading actual economic research before posting confidently incorrect Econ 101 oversimplifications. The "immigrants take our jobs" fallacy has been debunked by virtually every reputable economic study for the past 30 years. This isn't some fringe academic view. It's the overwhelming consensus of actual economists who study this for a living. Your intuition about "more workers = lower wages" seems logical but falls apart when tested against actual economic data. The real world is more complex than a supply-and-demand graph from an introductory textbook.
Nowhere in the economic research does it explain what you are so confidently stating, that wages are, somehow, the only thing in all of economics where positive supply shocks do not matter. The arguments that tend to be made are that, on a long enough time horizon, mean wages increase because overall economic output goes up, and mean economic output goes up because there is a higher supply of available labor.
You're fundamentally misunderstanding both the economic research and my argument. No one is claiming wages are "the only thing in economics where positive supply shocks don't matter."
The research shows that labor markets aren't simple supply-demand curves because of complementary productivity effects and gains from specialization, selection effects, and, of course, demand generated by the immigrants. If you have general labor size increase, in general equilibrium with a responsive central bank interest rates will lower to keep employment tight.
This isn't about "long enough time horizons" - studies find positive or neutral effects in the short and medium term too. The fundamental issue is that your model assumes a fixed economic pie that immigrants simply divide into smaller slices, when in reality immigrants help grow the pie overall.
Okay. Which other things don't see effects from positive supply shocks? You're just restating my premise about time horizons. The pie grows, with time, if a bunch of other things happen in the right order. Wages haven't grown in two decades in the UK, your original example. So, how are you defining short and medium term? Three decades?
man idk, maybe it was the long conservative rule after the crash? Maybe it was the long austerity? Maybe it was the huge mass of natives that voted to crash out of an agreement with a bloc that handles over 80% of their trade?
More seriously...
- for US: The newest NBER IV estimates put the wage effect of all 2000-19 US immigration at +2 % for non-college natives. Show me a UK study of similar vintage that finds anything near –2 %.
- for UK: UK real wages tracked productivity one-for-one after 2008; BoE and NIESR pin that on capital deepening, Brexit and austerity. Not on immigration, which the MAC finds moved wages by _at most_ –1% (aggregate, not yearly!) and the final report was ~0.1%, basically a null finding.
- We've already been through lump of labor, so I don't know why you've been banging on equilibrium.
And to finally address your time horizons: Short-run? Mariel-style shocks still show null effects. Medium-run? 2009-20 UK data flips positive. Long-run? Productivity wins. Pick your horizon. Immigration is at worst a rounding error next to TFP, which is positively associated with migration.
Happy to dive deeper, but at this point the burden of proof is on anyone claiming large negative wage effects. The best evidence, across multiple methods and countries, just isn’t there.
Got a link to that NBER study? It's not that I don't believe you or think you're making it up, but I would like to understand what instrumental variable they were using to make the claim that an increase of low skilled immigration makes wages for non-college educated natives go up.
I skimmed this and read a few sections closely. Most econometric papers spend too much time rambling on about stuff that most people who actually seek out and read these things already know. Anyway, +2.6% in wages for non-college educated natives, over 20 years, is not a ringing endorsement. I honestly doubt anyone would notice this, and over two decades that's just noise, frankly. If I'm understanding one of the central claims correctly, it's not that low skilled immigrants "took" jobs from natives, it's that natives ended up working more, and as a result their earning power increased. That's ... also not a ringing endorsement. Sure, you didn't get fired in favor of an immigrant, but you had to take a second shift to stay in the game. Wow what a benefit.
I'm also going to quibble a bit with how they constructed this Instrumental Variable. The way it's constructed, the higher the turnover (for lack of a better term) from native to foreign, the better the predictor it is (because in their theory, it's exogenous to wages at the local level). Does anyone really believe this? Immigrants respond to incentives just like anyone else. They're not choosing to immigrate to Sac City, Iowa, or any other low-COL/low-wage town, for a reason. It also doesn't answer my original question, which was central to my initial claim: why? The "why" is going to answer a lot of other layered questions about hostility to ever-increasing immigration. Are firms moving in that exploit low-wage immigrants, generating other adjacent economic activity? Probably! Not controlled for or referenced at all in this, and this paper is by no means a definitive conclusion that high low-skill immigration is good (even on a two decade timeline, which even typing that is just absurd).
The +2.6% is hourly pay, not extra hours; it’s the difference between flat wages and one more year of raises. Every credible quasi-experiment, from Denmark’s refugee lottery to U.S. enforcement crackdowns, confirms that more immigrants leave natives at least as well paid, often better off, because firms invest, prices fall and natives climb the job ladder. Shift-share IVs have been combed over by three separate methodological papers and pass; drop them and refugee lotteries STILL give you the same answer. UK stagnation is a productivity story (zoning, anyone?), not an immigration one. So unless you have a better identification strategy that overturns all of these results, the weight of the evidence says immigration grows the pie, and natives get a slice (if you can't believe any synergy than at least just bearing a smaller share of defense spending).
I'm also going to flip it around for a second. As I said with the Mariel boatlift study, where a 7% increase in the labor force yielded more or less no impacts on hyperlocal labor force, even considering (possible? I know little havana right now is mostly spanish but idk what it was back then) language and skills barriers. How do you explain that? That is the most short term of local supply shocks with basically no short term employment or wage impact. Thirty-five years of re-checks (Card 1990 → Borjas 2015 → Clemens-Hunt 2019 → Peri-Yasenov 2019 → Lewis et al.) still show more or less zero effect on native wages or jobs (once you fix compositional glitches in Borjas’s sample). If a shock that extreme can’t push wages down, the `more workers = lower pay` story is busted.
Yeah I got that part, maybe you're misreading. You seem to be interested in advertising that you've read all this stuff, which is great, a lot of people have. But you aren't really addressing the problem statement, so I'll state it clearly: no one gives a shit about a 2.6% hourly wage increase over 20 years. They don't care because immigration comes with a bunch of other externalities, that are very near term, that academics deliberately remove from their models. "Natives get a slice ... of 1.7-2.6% ... over the course of 20 years" is not a convincing argument, which is why most of the quantitative debate about immigration is largely academic. If the benefits were so obvious we wouldn't need a team of nerds to tell us that, well, actually your hourly wages do go up ... eventually. Insofar as you care about some sort of policy outcome here, you are going to have to figure out a different way to frame this.
I'm not familiar with zoning laws in the UK to comment, so, sure. Maybe a byzantine zoning bureaucracy is the problem there, that does ring distinctly "British" to me.
I haven't read the Mariel study, and honestly I don't really have any interest in it because the underlying story is that Cubans just replicated their own economic structures in a hyper-contained locality, with significant ethnic solidarity given a shared history of hardship. Again, there's qualitative aspects to this that economists - especially the econometricaly inclined - struggle with.
> Cubans just replicated their own economic structures in a hyper-contained locality, with significant ethnic solidarity given a shared history of hardship
Damn if only there was a way to study assimilation. Wait, there is, we have, and if you look a few replies up you'll see that its basically a complete success with sufficient NGO support that vastly boosts social participation.
> If the benefits were so obvious we wouldn't need a team of nerds to tell us that, well, actually your hourly wages do go up ... eventually.
This kind of thinking leads to Trump. Unironically. Handwaving about "if it were real I'd know of it" is what leads to terrible economic policy.
Well, if it were real people wouldn’t have voted for Trump. What you’ve presented is, like I keep saying, the most unconvincing tidbit of minor benefits. You seem totally uninterested in addressing the problem, that no one cares about this study and what it says because it’s such a tiny effect on the margins, utterly impossible to translate into daily life.
I’ll keep making whatever claims I want, and you can keep gatekeeping (or attempting to) as much as you like (now I really won’t read the Mariel study, nevermind that you are conditioning success on Uncle Sam handing over money to make it work). The force with which state something as plainly obvious only appears as such inside the spreadsheets, so enjoy them.
spite driven willful ignorance is something that I didn't expect to find when starting this conversation.
I'm mostly looking for you to retract your claim about how short-run supply shocks must obviously show up in wages and employment.
EDIT: also nowhere does it require fiscal outlays for assimilation, the single biggest thing is expedited provision of work permits, which is obviously fiscally positive.
Strawman. You're talking about wages and jobs in aggregate which wasn't my argument at all. Nothing you said addresses how my salary is affected in the industry the person is joining.
Sure, in total, other jobs may be created and growth is increased -- it's essentially a tautology.
I don't know why you think you're entitled to any particular job due to government policy, that seems like a really poor and inefficient way to run an economy.
This is how I think about unlimited data plans haha. I think a rate limit is easier to stomach (e.g., X requests per hour where they bank up to one hour so you can burst up to 2X for especially crazy hours or something).
I noticed if I need something hard to find I have to do a dozen+ different queries (and sometimes not find anything because it doesn't exist). Both with Kagi and Google the result is the same but with Kagi I also rack up a bunch per one attempt to find something. And if I need something easy to find but lazy both Google and Kagi reliably show the first correct result.
So it's either unlimited or nothing. But since I know Google's search operators well I don't have trouble finding things if they exist so $10 per month is hard to justify. Plus, you're anonymous with Google but you're not anonymous with Kagi since you pay them.
But Kagi can be good for tech illiterate relative you want to shield from sus sites.
> Plus, you're anonymous with Google but you're not anonymous with Kagi since you pay them.
The idea that you're anonymous with Google is laughable. The amount of data they aggregate is well known. Their entire business model is to know who you are.
Maybe before enforcing such nonsense, they should standardize function docs like javadocs. How are you supposed to know which errors to "except" if it's not documented anywhere anyway?
The battery life is what got me off. I don't have good charging habits, and nothing was worse than not being able to use your activity tracker for your workout because you forgot to charge it. I got a Garmin and I hardly ever have to think about it now.
Having to plan your day around when to charge your Apple Watch doesn't sound like a good experience to me.
But I'll start with the simple version of the critique. Individual commuters make their decision about whether to make a trip based in part on the time that trip will take (a function of road congestion). Thus, adding road capacity will temporarily lower the cost of a trip, allowing more people to make it until congestion reaches the pre-expansion level. Housing does not have this problem because consumers make decisions about where to live much more infrequently.
As you can tell, "induced demand" is a bad if rather catchy name. To be fair to urban planners, the throughput of an expanded road does increase. However, the experience of using it stays the same or gets worse, and the cost (both $ and space) is disproportionate. Is the pro-transit argument, which I subscribe to.
Of course, the real world includes large real estate firms which are making frequent decisions on housing stock. So I agree that the build-more-housing-and-everything-will-improve crowd is wrong. But I don't think YIMBY/transit and anti-transit are the only choices. The former is a common archetype these days.
I can't imagine why people use Airflow these days. Its DAG DSL means you have to fit your biz logic to their paradigm. I expect a framework to fit nicely with existing biz logic. And that means you're essentially stuck. It has no interoperability; hence the need to build custom operators for everything.
It doesn't scale for teams because the package is incredibly bloated, so once you need to run multiple images via K8s operators, you lose out on a lot of other Airflow functionality because it all assumes you have your biz logic embedded within DAGs, but at the same time I don't know why anyone would develop this way.
DAGs are fine. Their DSL is not because it's abstracting the wrong things in the wrong place. It's a global file with static definitions; why do you hardcode KubernetesOperator when maybe you don't want a KubernetesOperator in a test env? There is also no type safety between tasks/operators. And it's an extremely dependency heavy package with no client/server isolation so bundling Airflow for multiple teams is just not viable.
Why do you think you define DAGs in Python? The point is to be dynamic, exactly to do things like switching between operator types based on things like the environment. Sorry you really don't seem to know a lot about airflow for your strong opinions against it, I'm out, no offense intended at all.
I'm well aware that's "possible", but if you have to build your own abstractions and CI/CD to make it usable this way, it doesn't seem very well designed.
> I can't imagine why people use Airflow these days.
You lose out on a lot of other Airflow functionality because it all assumes you have your business logic embedded within DAGs.
As an old Airflow user, I can relate with that.
The contributors are great and the speed of the new features and fixes are great also.
I see two main issues with Airflow: (i) the lack of good interoperability with a cloud native stack, specifically with the k8s operators, that sounds quite hackish and (ii) lack of a better version control in the DAGs in way that we can have actually real lineage.
The point is that for most of the veterans in ~ETL~ Data Engineering in Cloud Native stacks, we just want to send a cli command or a Make command to a container to execute the thing that we want.
Apache Airflow does that, but as most of the things in the Python ecosystem at the time that you have a great UI (not confuse with UX) become way easier to embed business logic in the DAGs and have your application embedded with Airflow logic.
I will never bash Airflow since I got a lot of money in consulting migrating from and to it, but the general feeling of working on a daily basis for almost 7 years made me realize the sad reality around orquestrators. In the past, those used to be kind of the bedrock technology that you would never want to migrate, but today, due to this feature bloat and lack of vision, I started to see orquestrators as disposable as Java Script frameworks.
Prefect has more polish and is easier to get started than any of the existing options. We've been running their self-hosted for over three years and it basically stays out of the way.
We looked at Dagster as well as Airflow. I really, really liked Dagster but the BI team didn't.
I cannot imagine using Airflow for anything meaningful and respecting myself at the end of a work day. The local development experience was abysmal. Deployments sucked.
That being said, if you're not using anything except maybe cron right now, and if you don't care about the solution being a proper data pipelines orchestration platform trademark symbol, I'd recommend starting with Windmill.
I've done a lot evaluation of such frameworks, and I hope to publish more on it. It really depends on your requirements. I would look at Prefect, Flyte, Dagster, and Temporal ahead of Airflow, though.
Airflow is kind of the default platform. If you do not have a goto alternative which is superior in multiple dimensions today, I think that says the problem itself is hard.
People use it because it comes with a fully functional web UI.
(That's really the only reason why it's used. If it was just ad-hoc bash scripts underneath the UI I don't think anybody would mind.)
> I expect a framework to fit nicely with existing biz logic.
I wouldn't, that's just describing a custom codebase. Airflow comes with bells and whistles that can be taken advantage of if you fit your execution in their DAG model, that's all. Those bells and whistles can be an excellent pattern to work with. You can go all in on operators, or you can just have an operator call out to an external task that does all the work and only rely on Airflow for its retry/alerting mechanism while keeping the external task in language of your choice.
Airflow owning the scheduling, retry, task branching/dependencies is fine. But a few issues: to take advantage of Airflow's core features (XCOMs, task mapping, etc) your biz logic/tasks must be in the same venv, which is not sustainable as teams grow. Once you have external tasks, you now have a more complex system to operate and test, and your Airflow installation is now very overkill (you need a pool of workers just to monitor external... workers?).
You don't need to have your biz tasks in the same venv as your airflow worker to take advantage of task mapping or XCOM.
You just need a well defined interface on how to call your biz tasks, and then you can use any of ExternalPythonOperator, DockerOperator, DockerSwarmOperator, KubernetesPodOperator, etc. etc. or write your own to pass in values or data to your task however you want.
Airflow is quite complex, and I don't recommend it as people's go to, but IMO that's in large part because it is so unopinionated about how you call and run your tasks and leaves the configuration up to you. But this also means it ends up being a lot of people's choice because they are able to get it to fit what they need.
> hence the need to build custom operators for everything
But isn't that the point? I never got the impression that you were supposed to build everything with the built-in operators, they are just the "batteries included" part you can wrap or extend. I really don't understand the criticism.
My criticism was always xcom, but that's a moot point now that we have TaskFlow. Airflow is awesome and very flexible you just have to adapt to how it's supposed to be used instead of fighting against it I find.
That might be fine for small teams, but any 100+ person company would already have their own abstractions and don't need to reinvent those wheels just for Airflow. But even if you're smaller I still think you're setting up for failure if all your biz logic is within Airflow unless you are careful about making it reusable/shareable for other contexts. Eg, what if you want to convert some scheduled pipeline to some event-driven architecture with other systems? That means needing to refactor everything out. It's not interoperable or modular and that's why it should be avoided.
But in the core it is built around data pipeline concept, event driven pipeline will much more fragile. Airflow intentionally doesn't manage business logic, it works with "tasks".
Yes, but that means you are forced to build EDA on top of Airflow, which may not be ideal for many cases. You are stuck managing your pools/workers within Airflow's paradigm, which means all workload must (a) be written in Python and (b) have Airflow installed on the venv (very heavy pkg) and (c) be k8s pod or Celery (unless you write your own).
We only use KubernetesOperators, but this has many downsides, and it's very clearly a 2nd thought of the Airflow project. It creates confusion because users of Airflow expect features A, B, and C, and when using KubernetesOperators they aren't functional because your biz logic is separated. Eg., if your biz logic knows what S3 it talks to in an external task, how can Airflow? So now its Dataset feature is useless.
There are a number of blog posts echoing a similar critique[1].
Using KubernetesOperators creates a lot of wrong abstractions, impedes testability, and makes Airflow as a whole a pretty overkill system just to monitor external tasks. At that point, you should have just had your orchestration in client code to begin with, and many other frameworks made this correct division between client and server. That would also make it easier to support multiple languages.
> Airflow, in its design, made the incorrect abstraction by having Operators actually implement functional work instead of spinning up developer work.
> By simply moving to using a Kubernetes Operator, Airflow developers can develop more quickly, debug more confidently, and not worry about conflicting package requirements.
> Airflow lacks proper library isolation. It becomes hard or impossible to do if any team requires a specific library version for a given workflow
> There is no way to separate DAGs to development, staging, and production using out-of-the-box Airflow features. That makes Airflow harder to use for mission-critical applications that require proper testing and the ability to roll back
> Data pipelines written for Airflow are typically bound to a particular environment. To avoid dependency hell, most guides recommend defining Airflow tasks with operators like the KubernetesPodOperator, which dictates that the task gets executed in Kubernetes. When a DAG is written in this way, it’s nigh-impossible to run it locally or as part of CI. And it requires opting out of all of the integrations that come out-of-the-box with Airflow.
Airflow is far from perfect, but I don't understand your concerns. I work in a big and messy company and even messier department. We have jobs running in Databricks, Snowflake, sometimes we read data from API end points, or even files uploaded to SharePoint (my group is not building DW). Airflow lets me organize it in a single workflow. At least I know that every failed job is reported by email and I don't need to search multiple systems - all starts from Airflow.
> Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server?
Webserver is mostly UI. Scheduler service triggers the jobs.
We have groups which run everything as Bash Operator, no dependency issues that way.
You maybe have a very specific use case in mind, the main points of using Airflow for me
* Single orchestration center: manual job control (stop, pause, rerun), backfill; automated scheduler/retry; built-in notification
* Framework built around "reporting period" - it enforces correct abstraction, if a data batch is broken, I can rerun it and rerun all dependent downstream. How do you fix data in event driven workflow?
* managing dependencies
In most cases all Airflow does is running your job with passing it "date" parameter. You can test your code without Airflow - just pass it a date and run from command line.
If you need to convert a scheduled pipeline into some event driven architecture then yes, it will need a rewrite. Is there any case in which this wouldn't be true? What does it mean to be "interoperable"? Airflow drags can be triggered by events or they can trigger events if needs be. I admit it is not designed to do event streaming though.
There are many ads on Instagram that makes me wonder how this is legal; for example, I saw a fake site that pretended to be Marine Layer where the clothes were suspiciously cheap. I reported the ad, but they dismissed it.
And don't get me started on all the "mushroom alternative", "caffeine alternative", drugs. And it's like everyone and their grandmother has their own Viagra or Hair Loss drug company now
Google got hit for internet pharmacies and forfeited all the revenue they got.
I’m sure Google had no problem paying the same number of dollars in 2011 but really liked that money in their earlier days when there wasn’t as much big $ legit ad demand.
Its not just youtube. I unsubscribed from LA times after their email newsletter had one too many "Doctors don't want you to know about this!!!!" esque ads. Pure poison marketing. I can't believe people greenlight this crap to run on their platform. Maybe they already fired who greenlights ads.
For me Washington Post and Daily Beast newsletters were filled with ads for "Grunt Style" brand qanon clothing, sometimes all 5 or 6 of the ads in a given newsletter email. I ended up DNS blackholing some FQDNs, but I suspect that Grunt Style was targeting ads to "own libs".
There's a sucker born every minute. I suppose that when Googlers look at their huge paychecks it's easy to rationalize building tools that help scammers steal from suckers. From a legal standpoint they have plausible deniability but most YouTube advertising is just so slimy, like even more unethical than TV infomercials.
reply