Linguistics does draw the distinction between syntactic "tense" and semantic "time", but in that case English modal verbs wouldn't reflect "moods" at all, just "modality". They're all periphrastic. The same goes for perfect aspect, also periphrastic in English, though I don't know offhand how (or whether!) there is a terminological difference between syntactically-marked aspect and semantically-present aspect.

The same objection would theoretically apply to voice, where the English passive voice must be periphrastic too, but in that case everyone agrees that this is a distinction of voice and the difference between inflection (where grammatical meaning is expressed by changing the form of a single word) and periphrasis (where grammatical meaning is expressed by combining multiple words) isn't relevant. This is just an inconsistency in modern theory, which probably arose because voice isn't relevant to semantics at all.

Ignoring the modal auxiliaries, English would still have moods, subjunctive ("We demand that Robert be ejected from the book club") and irrealis ("If Robert were to be ejected from the book club, ..."), but neither of those is in a particularly robust state in the modern language.

Inflectional constructions express tense through morphology and periphrastic constructions express tense through syntax. Together they constitute expressions of grammatical tense.