DerinRuya

PROJECT DETAILS

I built DerinRuya from scratch as a Turkish dream-symbol archive designed to earn ad-supported organic search traffic. The product goal is straightforward: serve pages that directly answer a visitor's dream-symbol query, keep the archive growing continuously, and turn that traffic into ad impressions. The real engineering, though, is less about the front end and more about how the content itself gets made: every article on the site is written and published by an AI automation service that runs continuously in the background on the Linux server, with no human in the loop. Technology Stack and Architecture The front end runs on Next.js 16 (App Router) and React 19, with content pulled live from MySQL and rendered server-side. A separate Express 5 API handles the admin panel, authentication, and content endpoints. In production, Nginx acts as a reverse proxy, routing everything under '/' to Next.js and '/api/admin' and '/api/content' to Express; Next.js, Express, and the automation service each run as independent systemd services on Ubuntu. The Topic Pool: From 27,000 Queries to 4,000 Unique Titles Before content generation could start, I wrote a selection script that processes roughly 27,000 rows of SEMrush-sourced keyword data. It normalizes and dedupes rows split by difficulty band (easy/medium/hard), strips out unsafe patterns (drugs, sexual content, abuse, suicide, graphic violence) with a regex-based filter up front, clusters the remainder by stemmed root keywords, and picks the highest-volume representative from each cluster. What comes out is exactly 4,000 unique, non-overlapping dream topics ranked by difficulty and volume — the entire queue the automation pipeline draws from. Content Automation: An AI Service Running Continuously on Linux This is where most of my engineering time went. 'server/worker.mjs' is a Node.js process defined under systemd with 'Restart=always', so it never stops as long as the server is up; unlike a classic crontab line, nothing external triggers it — it drives its own heartbeat with a 30-second 'setInterval' and runs an "automation tick" on every beat. Each tick does the following, in order: 1. Creates the day's publishing slots: based on the schedule in the 'automation_settings' table (Europe/Istanbul timezone, seven fixed daily times — 07:00, 09:30, 12:00, 14:30, 17:00, 19:30, 22:00), it opens rows in 'publishing_slots' for whichever of today's times haven't passed yet. If the worker restarts mid-day it doesn't backfill past times in bulk — it only fills the remaining slots. 2. Assigns a topic to each open slot: it matches every pending slot to the highest-volume, easiest-remaining keyword cluster in the queue inside a database transaction, so the same topic can never be claimed by two slots at once. 3. Generates the content: for each slot it calls a language model through the OpenRouter API, constraining the response with a strict JSON schema ('response_format: json_schema', 'strict: true') to produce an 800–1800 word dream interpretation. Every generated article fits a fixed document shape: a title, a short excerpt, 4-5 concrete sub-scenario variants, and three separate interpretive lenses (Islamic/symbolic, psychological/self-reflective, astrological). Before the generation call, a separate "keyword assessment" step runs first — the raw query is distilled down to a clean focus phrase (e.g. "seeing someone throw a stone at you in a dream" becomes "seeing a stone being thrown in a dream"). 4. Filters for safety before generation happens at all: that same assessment step marks a cluster "rejected" and pulls it out of the pipeline entirely whenever the core topic is drug use, explicit sexual content, abuse, incest, graphic violence, self-harm, or suicide. Death, accidents, or a merely unsettling image are not by themselves grounds for rejection — the rule was designed to keep symbolic, non-graphic, non-glorifying dream narratives eligible for generation. 5. Checks uniqueness automatically: every new article's headings are stemmed and compared against the last 20 published articles' headings using Jaccard similarity. Anything that crosses the similarity threshold — including generic template titles like "What Does Seeing X in a Dream Generally Mean?" — gets rejected, the rejection reason is fed back to the model, and the same topic gets up to three more attempts. This step is the actual mechanism that keeps "the same template with the keyword swapped" content off the site. 6. Publishes it: an article that passes validation is written to MySQL as a draft and attached to its slot; once the slot's 'publish_at' time arrives, a separate step automatically publishes the draft and stamps 'published_at'. Fault Tolerance and Observability If a transient error occurs during generation (timeout, rate limit, model routing failure), the slot is retried with exponential backoff (starting at 30 seconds, capped at one hour); after three attempts it's still failing, the cluster is marked "failed" and pulled from the queue so the next eligible topic can take its place. There's also a dedicated recovery step that recognizes OpenRouter's transient model-routing failures ('404: No endpoints found') and automatically returns clusters rejected for that reason back into the pool. Every tick's outcome (articles generated, failures, publishes) is logged to an 'automation_runs' table, and a "heartbeat" row tracks the last run time, so whether the worker is actually alive is visible instantly from the admin panel or via 'journalctl -u derinruya-worker'. SEO and Visibility for AI-Driven Search Every generated dream page automatically emits Article, BreadcrumbList, DefinedTerm/DefinedTermSet, and FAQPage structured data derived from its visible sections; meta title, meta description, canonical, and index/noindex fields are managed per article. The sitemap is generated live from MySQL. A '/llms.txt' file and a matching robots policy were added so generative search engines and LLM-based assistants can crawl the site correctly — the goal being that traffic can come from AI-driven search surfaces, not just classic search. Conclusion DerinRuya's business goal is simple — organic traffic that converts into ad revenue — but the engineering behind it wasn't: it took a selection pipeline that reduces 27,000 raw rows to a safe, unique set of 4,000 topics, a scheduled publishing calendar that runs unattended seven times a day, a pre-generation content-safety filter, an automatic pre-publish uniqueness check, and a fault-tolerant Linux service to hold all of it together. The result is a site that writes and publishes seven new dream interpretations a day without me touching it — a self-sustaining content operation.