Showcase
My job scraper, rebuilt with AI to read every job for me and keep track of every reply

My first job scraper saved me a ton of time, but I was still opening every posting to find out what the job really was. Plenty of "remote" jobs turned out to be hybrid, a lot of mid-level titles were really senior roles, and LinkedIn hides the direct apply link when you aren't logged in. On top of that, replies from companies were getting lost in my inbox. So I rebuilt the whole thing around AI, and now it reads every job for me, finds where to apply, and keeps track of every response.
In 2024 I built my first job scraper to pull public listings from LinkedIn into a dashboard so I could skip all the sponsored posts and fake listings. It worked, but keyword filters can only do so much. A posting titled "Software Engineer" might want ten years of experience, be built entirely in Java, or require three days a week in an office halfway across the country, and the only way to know was to read the whole description myself.
At work I had been building AI-powered tools, and I realized that reading job descriptions is exactly the kind of thing an LLM is great at. So I decided to rebuild the project from the ground up as a full AI pipeline. Every day it finds new jobs, has AI throw out the ones that don't fit me, looks up the direct link to apply, and later on even reads my inbox to see which companies wrote back!
I kept Python and BeautifulSoup for gathering jobs, but almost everything else changed. The searches and filters used to be hardcoded in my script, so I moved them into the dashboard where I can add a search, skip a company, or change a filter without touching any code. LinkedIn's remote filter also doesn't work when you aren't logged in, so the scraper reads each job's description to decide if it's really remote, hybrid, or in person. It also pulls out the salary range and cleans up postings that have closed.

One of my favorite additions is the apply link lookup. LinkedIn hides the external apply link from guests, so for every new job the scraper checks the public job boards of eight different hiring systems, like Greenhouse, Lever, Ashby, and Workday, and compares the descriptions to make sure it found the right posting. Now most jobs have a direct link to apply on the company's own site. I also moved the database from SupaBase to PostgreSQL, and the scraper runs on a schedule every day on Railway instead of my RaspberryPi.
This is the part that saves me the most time. After the keyword filters run, AI reads every new job alongside a summary of my background and a short list of rules, like the job has to be fully remote, mid-level, and built mostly in a language I actually work in. I used structured outputs so the AI always answers with a simple yes or no and a one sentence reason, which makes the results easy to save and easy to check. Jobs that aren't a fit get removed before they ever show up in my dashboard, and if a posting is unclear on a rule, the AI keeps it so I don't miss anything good.

Getting the rules right took some trial and error. Whenever I dismissed a job by hand, that was a sign the rules missed something, so I tested my changes against the jobs I had already dismissed and the ones I had applied to. A good rule had to catch the first group without throwing out any of the second!
Once I started applying to more jobs, keeping up with replies got messy fast. I built a second job that runs every afternoon, reads new mail from my inbox, and has AI decide if each email is a reply to one of my applications, like a rejection, an interview request, an assessment, or an offer. When the AI can match the reply to a job, the job moves to the right status automatically, and anything it can't match waits on a Responses page for me to look at.

I also added a timeline to every job that shows when I applied, each reply I got, and every status change along the way. Now I can see the whole history of an application in one place instead of digging through my email.
I rebuilt the dashboard with Next.js 16, React 19, and TypeScript, using shadcn/ui components, TanStack Query and Table for the data, and Drizzle to talk to the database. The jobs page has filters for posted date, status, location, and apply method, plus a search bar and quick stats at the top. Clicking a job opens all of its details in a side panel, including a suggested salary to ask for based on the listed range, and a place for notes like recruiter contacts and interview dates.

I also added charts so I can see my search over time: how many new jobs came in each day, and how many I applied to, heard back from, or dismissed. It's really motivating to see the progress, and it shows me right away if the scraper ever stops finding jobs.

This project taught me a lot about putting an LLM to work inside a real pipeline. The prompt is only part of it. Clear rules, structured outputs, and testing changes against real data are what make the results something I can trust. I also learned to keep an eye on cost the hard way: one test run of the inbox check on a big backlog of old emails drained my API credits in one go! Now I always try new features on a small batch first.
Most of all, this is the tool I actually use every day during my job search. It went from a script that saved me some scrolling to a pipeline that does most of the reading for me, so I can spend my time applying to jobs I'm genuinely excited about.