Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Unrelated to Python, but is anyone aware of a tool like Make that can handle file path wildcards in prerequisites?

For example, let's say I have a rule that consumes *.json to produce some target. I want it to rebuild that target if any *.json files are added or removed (or modified).

As far as I'm aware, Make relies on file system timestamps, so even if it supported wildcards (I'm not sure if it does or not), it wouldn't be able to notice that foo.json has been deleted and a rebuild of the target is needed.

I thought I'd ask here before I go build such a tool (to support incremental rebuilds of a static site).

Edit to add that my particular use case has a non-prescriptive set of directories, nested a few levels deep. I'm actually realizing that that is probably a bigger hurdle for using something like Make (e.g. transform all *.md files, no matter how deep; but copy everything else, like images, verbatim; oh, and also then aggregate all *.md files into an Atom feed). Yes, I know this is asking a lot of something like Make!



Did you try searching stack overflow? https://stackoverflow.com/a/43402649

If you dedicate a specific directory for all your JSON, then you can depend on that directory and it will do what you're asking for.


I did spend quite a bit of time researching approaches, but I didn't come across this particular idea. I'll give it a try. Thanks!

Note: in my case, I have a directory structure that's a few levels deep, with an non-prescriptive set of directories (one subdirectory per category, with no limit on the set of categories). Maybe Make handles directories better than I realized (I'd always seen it recommended to use a Makefile in each directory--something I want to avoid).


Here's one way to do this.

I have used this method (directory mod-time triggering, let's say) for a simulation-summarizer which analyzes whatever pile of simulation-output-files happen to be in a given directory. If you run a new simulation, the directory changes and the analysis is re-done by running "make".

I used the Gnu make $(wildcard ...) for the template expansion, instead of using shell expansion. This is to take care of the no-file case, so that jsons/*.json will expand to nothing rather than to the literal jsons/*.json (which does not exist).

  $ cat Makefile 
  file.out: jsons
         cat $(wildcard jsons/*.json) /dev/null > file.out
  
  
  $ ls -R
  Makefile  jsons/
  
  ./jsons:
  foo.json
  
  $ make
  cat jsons/foo.json /dev/null > file.out
  
  $ make   # no file mods => no-op
  make: `file.out' is up to date.  
  
  $ touch jsons/bar.json
  
  $ make   # new file => re-make
  cat jsons/bar.json jsons/foo.json /dev/null > file.out
  
  $ make     
  make: `file.out' is up to date.
  
  $ rm jsons/foo.json 
  
  $ make  # deletion => re-make
  cat jsons/bar.json /dev/null > file.out
  
  $ rm jsons/bar.json 
  
  $ make  # nothing there
  cat  /dev/null > file.out
  
  $ make
  make: `file.out' is up to date.


Definitely avoid makefiles per directory, see Recursive Make Considered Harmful https://accu.org/journals/overload/14/71/miller_2004/

The approach being recommended in the sibling comment to this is quite nice!


Thanks to the folks who replied! Looks like I didn't dig far enough into GNU Make. It's got a lot of the functionality I'm looking for (and I just didn't realize it). Hopefully I don't need to use it, but it even has an "eval" function for dynamically generating its own input.


GNU make at least can do that.


Interesting! I had originally tried to do this with GNU Make (with the most naive approach you can imagine) and it wasn't able to notice that a prereq that matched the wildcard had been deleted (i.e. it thought the target was up to date, when I wanted it to rebuild).

I'll read up on the wildcard function and see if that is what I was looking for:

https://www.gnu.org/software/make/manual/html_node/Wildcard-...

Edit: A sibling comment also pointed out putting the wildcard'd files into a directory to help Make notice deletions. I'll give that a try first.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: