You would have an API that makes the query shape, the query instance with specific values, and the execution of the query three different things. My examples here are SQLAlchemy in Python, but LINQ in C# and a bunch of others use the same idea.
The query shape would be:
active_users = Query(User).filter(active=True)
That gives you an expression object which only encodes an intent. Then you have the option to make basic templates you can build from:
With SQLAlchemy, I'll usually make simple dataclasses for the query shapes because "get_something" or "select_something" names are confusing when they're not really for actions.
This is a better story because it has consistent semantics and a specific query structure. The db.getUsers() approach is not part of a well-thought-out query structure.
> It's the Thing module's responsibility to correctly set up the test doubles involved.
But why?
Maybe there's some fundamental difference of opinion here, because I'm thinking definitely not, no way, test-only code paths should never reach prod, and code that ships should never even know that testing is a thing that happens.
That's the trick - it is not test only code. You use the same objects, the same code.
The code has to be written so it takes in the altered unit via parameters or other DI or so it knows that the current system is set to short circuit some calls.
Ie, you could have MyClient(testHTTPClient).GetResource(foo) used in tests and MyClient(realHTTPClient).GetResource(foo). The testHTTPClient would get to the actual connection part and return a configured response or error.
Your entire logic is the same, your code could "receive" a forced 404 or timeout or 200 OK. It is up to the testHTTPClient that is only changing how the http connection is handled.
I call these unit-integration tests. You are checking all units work together while not actually working with the outside world.
You’re responding to a comment containing a method call `thing.makeThingDoerForTest(1, 2, 3)` and saying “That's the trick - it is not test only code.”
It goes a bit into the same direction rust went by shipping the unit tests usually inside the module. The thought is intriguing to keep the „mock“ setup up to date even when doing refactors. But I wonder if this solves the issue.
I tend to not use mocks. And I try to stay away from spies. These two tie the tests to close to the implementation like the article also described.
The reason why I don‘t ship test setup in production code is the simple fact that the test / spec should not be based on production code. Hard to put in words for me but the test should be impartial. It sets up its test world and run the subject in it. It should not use the subject to describe the test environment. I think this will lead to false positives.
I have an example where a coworker used constants from production to assert return values. I stated that the test can‘t use them because the test is the specification and needs to be in control what is correct and what is not. Even if that means we write duplicate string values. In this case somebody changed a const value to a wrong value. But for the test this was all fine.
What is the harm in a test only path in production? If you are an embedded systen with limited memory there is obvious harm but most code isn't that. There are other possible harms but they are less obvious and may not imply. There are also many different ways to implement this.
i did something like this where in production I have if function pointer isn't null return the result of it instead of running the production code (and an unlikely compiler directive on the if). The function itself is implemented in a test only shared library so the cost to production is low. If you can break in you can set that function - but if you can inject arbitary code we are already sunk.
`makeThingDoerForTest` is a standalone function (or static method) that never gets called in production code. I don't see why it needs to be in an entirely separate module. If you want, you could put it in a separate thing_test module, but then you'd have to expose a more powerful constructor for thing_test to access. I'd rather hide that inside the thing module, and I see no benefit to splitting this functionality across two modules just for the sake of not mixing my peas with my potatoes.
> no way, test-only code paths should never reach prod, and code that ships should never even know that testing is a thing that happens.
What's your reasoning for this?
As a counter-example, the hardware you are using right now almost certainly has test points throughout the circuitry.
Beyond all the uses the test points get during development and manufacturing,
they are essential for repair and re-calibration after being sold to the customer and put into production.
So much needless discussion because someone makes an ill thought out hot take. Most engineered objects of any substantial complexity include built in test systems. They would be impossible to build without them.
In-situ fault testing and isolation is an engineering principle that per pervasive across many domains. You want your auto manufacturer to rip anything related to tell you why it doesn't work?
You are just trying to defend a previous opinion by throwing out some acronyms to dress up your argument.
They should spin two boards, one with jtag and one without and then how would they debug the board they just removed the jtag on?
All systems are dynamic, all reliable dynamic systems use feedback, but you are arguing that engineers would rather remove those feedback mechanisms?
Honestly I'd probably resist this too. Maybe you could practice on me? Help see the benefit?
I'm seeing three main points in this advice:
1. Don't misuse mocks. (Mocks, fakes, whatever.)
2. Write testable, well-factored code with pluggable dependencies.
3. Don't use mocks at all. Use "nullables" instead.
I'm totally on board with (1) and (2) as good advice in general, no matter how you're testing. But (3) seems like an independent choice. What benefit does (3) specifically deliver that you wouldn't get with (1) and (2) and mocks instead of nullables?
I'm ready to admit I'm missing something, but I don't see it.
The beauty of the nullable approach is that when you're testing the system under test (SUT), you're testing it together with its dependencies. Not just that - you're running the real production code of the dependencies (including transitive dependencies!), just with a off switch toggled.
What the off switch does is really just prevent side effects, like making actual HTTP requests to the outside world or reading bytes from the file system. Using embedded stub makes this a breeze (another revelation for me).
For example, while building the "/v1/add-account" endpoint, you write a test for the request handler function. If you write this test using Shore's approach, you'll exercise the whole tree of dependencies, down to the leaves.
This is what is meant by sociable tests - the SUT isn't isolated from its dependencies. It's the real code all the way down, except that you're calling createNull() instead of create() to instantiate the classes. There are many benefits here, but to me the most important ones are: (1) you don't need to write mocks as separate classes, (2) you can catch errors that arise in the interplay of the various classes or modules and (3) as a result you get a lot more confidence that everything is still working after refactoring.
A sociable microtest is a little like an integration test, but you don't exercise any out-of-process dependencies (perhaps with the exception of the RDBMS) so the tests run in milliseconds rather than seconds.
You commented elsewhere that you're worried about the separation of test code and prod code. Yes, this is a bit of a holy cow that I also had trouble adapting to. It turns out, having test-focused code mixed in with your prod classes is totally fine. I'd perhaps prefer a cleaner separation but honestly it's not a big issue in practice.
I'm all aboard this, in fact it's how I like to write my "unit" tests, but still I'm unsure about this nullable and test code with prod.
What I do is I simply mock as little as possible (leaf nodes such as timers, remote calls, sometimes RDBMS/filesystem, ...). But I'm not sure what embedding the mock in the prod code gains you? I wonder if part of it is down to the language used?
For instance it says "Put the stub in the same file as the rest of your code so it’s easy to remember and update when your code changes.". Maybe it's because they're using a dynamic language? I'm using C++, so any change in interface will fail to compile the mock, making this less important. You could change the behaviour whilst keeping the same interface, but it's very rare in my experience (and definitely should be avoided).
Nullable I could see some values in some cases where you also want to be able to disable a functionality in prod. But in that case you have a function in the interface to turn it off, so you might as well use that. I can see where using Nullable in construction avoids having to expose this method through the whole dependency tree, but at the same time you lose the ability to dynamically turn in on/off.
I think the big benefit comes from hiding implementation details away from tests. Here—let me copy-paste an example I used elsewhere in this thread:
---
With mock injection:
m = new FooMock()
m.call("Bar", 1, 2)
m.call("Baz", 3)
p = new Thing(m)
actual = p.doSomething()
assert(expected == actual)
With a test-specific factory method:
p = Thing::makeTestThing(1, 2, 3) // static method
actual = p.doSomething()
assert(expected == actual)
With the test-specific factory, mocked dependencies are encapsulated by Thing. You can configure the their behaviour by passing different parameters to the factory method, but implementation details like `m.call("Bar", 1, 2)` are hidden. The test doesn't need to know that `doSomething` calls `Foo.Bar`.
I'm not sure whether the author of the article would consider `Thing` to be a nullable if `makeTestThing` uses a conventional, automatically-generated `FooMock` under the hood, but I don't think it really matters. To me, the big benefit comes from `Thing` assuming the responsibility to configure and inject its own mocks.
Thanks, this helped me understand some things. Unfortunately I don't have time to write a real reply right now, but I think I'm more convinced than before. Thanks for taking the time.
Part of testing is risk management. With mocks you run the risk of the behaviour of your mocks being out of sync with the actual systems you’re mocking. By moving the test-time behaviour closer to the runtime behaviour, you reduce that risk somewhat. If you have other ways of managing that risk or you just never see that happening, you’re good.
The value-add for (3) is that "nullables" encapsulate all the fiddly details that mocks (tend to) expose.
Here's an example. With mock injection:
m = new FooMock()
m.call("Bar", 1, 2)
m.call("Baz", 3)
p = new Thing(m)
actual = p.doSomething()
assert(expected == actual)
With a test-specific factory method:
p = Thing::makeTestThing(1, 2, 3) // static method
actual = p.doSomething()
assert(expected == actual)
With the test-specific factory, mocked dependencies are encapsulated by Thing. You can configure the their behaviour by passing different parameters to the factory method, but implementation details like `m.call("Bar", 1, 2)` are hidden. The test doesn't need to know that `doSomething` calls `Foo.Bar`.
I'm not sure whether the author of the article would consider `Thing` to be a nullable if `makeTestThing` uses a conventional, automatically-generated `FooMock` under the hood, but I don't think it really matters. To me, the big benefit comes from `Thing` assuming the responsibility to configure and inject its own mocks.
In theory it's a nice abstraction, and the benefit is clear. In practice, your repository likely ends up forwarding its arguments one-for-one to SQLAlchemy's select() or session.query().
That's aside from their particular example of SQLAlchemy sessions, which is extra weird because a Session is already a repository, more or less.
I mean, sure, there's a difference between your repository for your things and types you might consider foreign, in theory, but how theoretical are we going to get? For what actual gain? How big of an app are we talking?
You could alias Repository = Session, or define a simple protocol with stubs for some of Session's methods, just for typing, and you'd get the same amount of theoretical decoupling with no extra layer. If you want to test without a database, don't bind your models to a session. If you want to use a session anyway but still not touch the database, replace your Session's scopefunc and your tested code will never know the difference.
It's not a convincing example.
Building your repository layer over theirs, admittedly you stop the Query type from leaking out. But then you implement essentially the Query interface in little bits for use in different layers, just probably worse, and lacking twenty years of testing.
Thanks, that makes a lot of sense. I don't have a whole bunch of experience with SQLAlchemy itself. In general, I prefer not to use ORMs but just write queries and map the results into value objects. That work I would put into a Repository.
Also in my opinion it's important to decouple the database structure from the domain model in the code. One might have a Person type which is constructed by getting data from 3 tables. A Repository class could do that nicely: maybe run a join query and a separate query, combine the results together, and return the Person object. ORMs usually tightly couple with the DB schema, which might create the risk of coupling the rest of the application as well (again, I don't know how flexible SQLAlchemy is in this).
There could be some value in hiding SQLAlchemy, in case one would ever like to replace it with a better alternative. I don't have enough experience with Python to understand if that ever will be the case though.
All in all, trade-offs are always important to consider. A tiny microservice consisting of a few functions: just do whatever. A growing modulith with various evolving domains which have not been fully settled yet: put some effort into decoupling and separating concerns.
I've used SqlAlchemy in a biggish project. Had many problems, the worst ones were around session scoping and DB hitting season limits, but we had issues around the models too.
The argument for hiding SqlAlchemy is nothing to do with "what if we change the DB"; that's done approximately never, and, even if so, you have some work to do, so do it at the time. YAGNI
The argument is that SA models are funky things with lazy loading. IIRC, that's the library where the metaclasses have metaclasses! It's possible to accidentally call the DB just by accessing a property.
It can be a debugging nightmare. You can have data races. I remember shouting at the code, "I've refreshed the session you stupid @#£*"
The responsible thing to do is flatten them to, say, a pydantic DTO. Then you can chuck them about willy-nilly. Your type checker will highlight a DTO problem that an SA model would have slipped underneath your nose.
The difficulty you have following that is that, when you have nested models, you need to know in advance what fields you want so you don't overfetch. I guess you're thinking "duh, I handcraft my queries" and my goodness I see the value of that approach now. However, SA still offers benefits even if you're doing this more tightly-circumscribed fetch-then-translate approach.
This is partly how I got from the eager junior code golf attitude to my current view, which is, DO repeat yourself, copy-paste a million fields if you need, don't sweat brevity, just make a bunch of very boring data classes.
Just a heads-up if you haven't seen it: Overriding lazy-loading options at query time can help with overfetching.
class Author(Model):
books = relationship(..., lazy='select')
fetch_authors = select(Author).options(raiseload(Author.books))
Anything that gets its Authors with fetch_authors will get instances that raise instead of doing a SELECT for the books. You can throw that in a smoke test and see if there's anything sneaking a query. Or if you know you never want to lazy-load, relationship(..., lazy='raise') will stop it at the source.
SQLModel is supposed to be the best of both Pydantic and SQLAlchemy, but by design
an SQLModel entity backed by a database table doesn't validate its fields on creation, which is the point of Pydantic.
I can't take a position without looking under the hood, but what concerns me is "SqlModel is both a pydantic model and an SA model", which makes me think it may still have the dynamic unintended-query characteristics that I'm warning about.
I seem to recall using SqlModel in a pet project and having difficulty expressing many-to-many relationships, but that's buried in some branch somewhere. I recall liking the syntax more than plain SA. I suspect the benefits of SqlModel are syntactical rather than systemic?
"Spaghetti" is an unrelated problem. My problem codebase was spaghetti, and that likely increased the problem surface, but sensible code doesn't eliminate the danger
I mean that from the point of view of YAGNI for a small app. For a big one, absolutely, you will find the places where the theoretical distinctions suddenly turn real. Decoupling your data model from your storage is a real concern and Session on its own won't give you that advantage of a real repository layer.
SQLAlchemy is flexible, though. You can map a Person from three tables if you need to. It's a data mapper, then a separate query builder on top, then a separate ORM on top of that, and then Declarative which ties them all together with an ActiveRecord-ish approach.
> I prefer not to use ORMs but just write queries and map the results into value objects. That work I would put into a Repository.
Yep, I hear ya. Maybe if they'd built on top of something lower-level like stdlib sqlite3, it wouldn't be so tempting to dismiss as YAGNI. I think my comment sounded more dismissive than I really meant.
SQLAlchemy Session is actually a unit of work (UoW), which they also build on top. By the end of the book they are using their UoW to collect and dispatch events emitted by the services. How would they have done that if they just used SQLAlchemy directly?
You might argue that they should have waited until they wanted their own UoW behaviour before actually implementing it, but that means by the time they need it they need to go and modify potentially hundreds of bits of calling code to swap out SQLAlchemy for their own wrapper. Why not just build it first? The worst that happens is it sits there being mostly redundant. There have been far worse things.
The tricks you mention for the tests might work for SQLAlchemy, but what if we're not using SQLAlchemy? The repository pattern works for everything. That's what makes it a pattern.
I understand not everyone agrees on what "repository" means. The session is a UoW (at two or three levels) and also a repository (in the sense of object-scoped persistence) and also like four other things.
I'm sort of tolerant of bits of Session leaking into things. I'd argue that its leaking pieces are the application-level things you'd implement, not versions of them from the lower layers that you need to wrap.
When users filter data and their filters go from POST submissions to some high-level Filter thing I'd pass to a repository query, what does that construct look like? Pretty much Query.filter(). When I pick how many things I want from the repository, it's Query.first() or Query.one(), or Query.filter().filter().filter().all().
Yes, it's tied to SQL, but only in a literal sense. The API would look like that no matter what, even if it wasn't. When the benefit outweighs the cost, I choose to treat it like it is the thing I should have written.
It isn't ideal or ideally correct, but it's fine, and it's simple.
You seem to have stopped reading my comment after the first sentence. I asked some specific questions about how you would do what they did if you just use SQLAlchemy as your repository/UoW.
One could say the same thing about empathy, another "basic" skill.
Imagine how you'd rush to justify your lack of it here.
Everything you'd say in your defense would equally explain why someone wouldn't know the meaning of a certain unmarked number in the output of a random shell command.
You can't interrogate your qualia. A lot of people think (or really, feel) that makes them magic.
It doesn't.
But if you feel they're magic, you'll never believe that a "random" mechanical process, which you think you can interrogate, could ever really have that spark.
Turns out we have technologies and experiences with technology which weren't possible until very recently. Some things just look very different in hindsight.
Nothing changed in regard to so called hard problem of consciousness. All thought experiments and arguments like Chinese Room and p-zombies are still applicable.
Philosophy of mind hasn't started nor ended with Dennett, and definitely not with AI hype manufacturers.
Qualia might still be magic. Maybe we have souls or something, I don't know.
I'm comfortable with saying that this way of answering the question doesn't work, because the argument is simply, "How can it be false if I believe it's true?"
I subscribe to the ‘qualia is atoms in a trenchcoat’ school of philosophy personally, but I understand that it might be hard to accept and even harder to not be depressed about it.
LLMs lack the capacity for invention and metacognition. We're a long way from needing to talk about qualia; these things are NOT conscious and there is no question.
This website is absurd. "I don't think LLMs are all they're cracked up to be" "YOU'RE JUST MAD BECAUSE QUALIA AND YOU WANT TO BE MAGIC"
no the "magic" text generator just writes bad code, my dude
Only the people on r/singularity care about qualia in this context and let us remember this is a mechanism without even a memory
Nobody is thinking about qualia. That's only how I framed it. They just know the machine will never replace them. They're a person, doing person things, and it's a machine.
So every time it proves it can, they move the goalposts and invent a truer Scotsman who can never be replaced by a machine. Because they know it can't do person things. It's a machine.
The query shape would be:
That gives you an expression object which only encodes an intent. Then you have the option to make basic templates you can build from: ...where `exclude` is any set-valued expression.Then at execution time, the objects representing query expressions are rendered into queries and sent to the database:
With SQLAlchemy, I'll usually make simple dataclasses for the query shapes because "get_something" or "select_something" names are confusing when they're not really for actions.