Scraper of job listings directly from company websites. I found my last day job by using a scraper that visits company websites in search of job listings. Now I've turned it into an app for others to use and access jobs that are posted on company websites (rather than paid employer ads on Indeed or wherever). This gives the job searcher an advantage to find jobs not listed on job search sites and show the company you have taken time/interest to visit their site.
* Job listings for "Quality Assurance" and "QA" are split into different listings in Job Search.
* I really like the green highlight for Salary range! Personally, I would sort by jobs that list salary first, then by location (or relevance, or whatever).
* The filter was a little confusing to use. I see you talked about it with other users in here. It needs some love, but it's getting there. :)
* If you are going to target job searchers, it would be very helpful too see metrics based on the results. Here's a few examples I came up with
Example 1: I select Help Desk -> Chicago
I see a short-term graph showing whether demand has gone: up, down, or stayed the same - included is a red/green/yellow arrow giving me an idea at a glance. This helps me understand how many Help Desk postings are in Chicago
Example 2: I select Cybersecurity -> I also select Information Security -> NYC
I see a short-term graph showing demand for Cyber vs IS in NYC. This helps me understand if which job has more postings in NYC.
Example 3: I select Python Developer -> Boston & Dallas.
I see a medium-term graph showing demand for each location for Python Developer. This helps me decide whether demand is more consistent in Boston or Dallas.
Example 4: I select Asia & Canada -> Advertising (Under Industry)
I see a long-term graph showing the overall trend for that industry in each of those countries. This helps me track whether jobs are being outsourced, what I should expect in the coming years, and/or which country is the most competitive in that industry.
Yea there is a much better version of the search bar soon-to-deploy (which accounts for aliases like QA -> Quality Assurance) and it will match by word rather than the entire phrase (currently "software engineer" will not query "software test engineer"). Appreciate the callout here
You can find a toggle switch for "has salary" under the "other" filters which will show only those w salary, but good call perhaps that should be part of a sort feature (beyond just date)
The filters do need more love for sure. I like your examples for various metrics displayed in the UI. I did think it would be cool to have a Github-like array of squares that represent units of time with colors that show how it has been changing over time, would have to figure out how heavy of calculations those would be in real-time but I really like your idea here. Or a line chart might be better.
How did you design and build your homepage? I find that building the landing page and making it look like a professional, beautiful design is one of my biggest hurdles. I'm an experienced web developer but without a design to work off of - and especially accounting for mobile and dynamic sizing - I really struggle with this part of the work so I'm wondering what other people's workflows are for it.
Yea I feel ya, I’m definitely not a good designer based on how long this stuff takes me. I think we are all not that bad but once you spend 6 hours on a component you really feel your dev skills could be much better used. Anyway here are some of the things I used on the landing page:
Yes, I know, that's very easy to say, but in my experience:
1.) Imitating a good design can take a lot of developer hours. Many landing pages have fancy css effects and support many screen sizes and it can take quite some time to build.
2.) Finding a good designer is not easy. I have tried hiring on most freelancer sites with very poor results. The applicant pool is typically of low quality.
So I am wondering if others have found efficient/effective ways of going about #1 or #2, either by using certain tools or templates or by having a more clever hiring method.
Certainly creating a robust design that supports multiple screen sizes is no trivial matter. There are tools to make it easier but as with anything else, no single solution fits all developers and all projects. You say you're an experienced web developer and yet you find yourself deficient in this particular skill, so I'm not sure what to think. Maybe your strength lies more in back end programming?
In any case, I suggest looking into frameworks such as Bootstrap or Tailwind. Of course, there are also high-level solutions that don't require as much coding, e.g. WordPress, Wix, Squarespace, etc. WordPress plugins can make it very easy to apply fancy CSS effects without writing any CSS.
As for finding freelancers, I agree that's also a difficult task. I don't have any magic bullet there, other than to prioritize people with good, timely communication skills, and to avoid people on the low end of the price range since they will usually be less experienced.
I've been on both sides - working with developers for the past 5 years and leading design teams, so I get the frustration.
The key is finding someone who understands both design AND development constraints. Feel free to check out some of my work here: https://monadile.framer.website/ .Feel free to reach out via email: monadile.design(at)gmail.com or book a call via my website.
You could also do something in between - buy ready made templates and customize them. It is easier than mimic-ing and cheaper than hiring a designer. Something like https://tailwindui.com/
I would love a map of job postings to see where it might make sense to move to in the future. If there's 10 jobs within 50 miles... that might be a good place to buy a house.
Additionally, if I filter by 'north america' I still get jobs from canada and india because they're remote only. I would LOVE to be able to filter out those positions. Also I would love to be able to AND 'remote' and 'north america'. I would like to work remotely, but only for US companies
Thank you :) I appreciate the request + feedback. I have a story in the backlog to add location-specific links to the landing page but I really like your idea of having a map (heatmap or something) to show densities of jobs.
So the inclusive vs exclusive filtering is something that I struggle to perfect here. I'm tempted to throw both in the UI (since its ready to go on the backend) but its hard to explain to users. One thing you can do that is not so obvious is add a tag for "Canada" but click on the tag again which will put a line through Canada and exclude that location from your filter (still need to have helpers to show users how to do that). The 'remote' tag is probably the toughest one to parse of a job listing because it might appear anywhere within the text, so there is some inaccuracies for sure but its improving I hope!
Ah I could probably add filters for company locations specifically too (so you can filter US companies), that's an interesting use case too.
Thanks for the compliment too, it has been really fun to build
Oh that's a really interesting idea. Yea I dislike the idea of charging the job seeker but have not found a good way to monetize companies (not that they even know about me anyway)
What are you using to for the scraping? Playwright…selenium? I wanted to do something as a hobby but my IP kept getting reported lol. Also when you say companies…where are you getting the information from? Data brokers? Anyway, it is an interesting topic to me.
Selenium, although I'm using a wrapper library that uses it. I only query each company every few days or so which probably helps to not get banned IP-wise but also rotate them. But many of the company job links are through external sources too (lever, greenhouse, etc.) which don't seem to mind
The company data was gathered online for a long time until I found https://www.thecompaniesapi.com/ (which now is the source for much of that data)
I tried to use my own desktop machine to process some of these tasks. I can see my fans go jet mode when the scraping was being done lol. Do you have discord or any way to connect? Would love to chat around this topic. Feel free to drop any social media handles. I’ll ping you.
Ohhh yea I run into this memory issue very quickly when scraping (especially if you have a large URL dataset then it will inevitably find a website with a giant bit of markup). So I have to set timeouts and blacklist timely requests but also completely reset the (headless) browser on 2-3 requests (which is overkill but I am restricted on memory for those workers). Feel free to drop me an email sometime (should be on my HN profile)
The data is destroyed and no content from the web pages are reused or repurposed (each listing is merely a link + various tags that are created/associated upon viewing. My understanding is that public websites scraping is legal but repurposing their content might not be
Great site. Small feedback: There's a category 'Closure'-- I'm not sure if that's something I don't know about, but it definitely isn't for jobs using 'Clojure'.
Cool site - very clean lightweight interface which is great. Have a few friends that are looking that would for sure have checked it out had there been a free trial.
Thank you. Currently working on promo code functionality to give away to all the postings in Ask HN: Who wants to be hired? (December 2024) https://news.ycombinator.com/item?id=42297422
This is really cool. Do you have any interest in helping people auto apply to them? We can help you set it up with a really simple API call via Skyvern (https://github.com/Skyvern-AI/skyvern)
Scraper of job listings directly from company websites. I found my last day job by using a scraper that visits company websites in search of job listings. Now I've turned it into an app for others to use and access jobs that are posted on company websites (rather than paid employer ads on Indeed or wherever). This gives the job searcher an advantage to find jobs not listed on job search sites and show the company you have taken time/interest to visit their site.