Independent distributed/backend builder
Job Posting Skill Demand Analyzer Engine & REST API
Validated an asynchronous Python and PySpark engine on 500,000 real posting records: 411,135 canonical postings, 156,491 skill matches across 73 canonical skills, and 0.738ms p95 over 1,000 populated sequential warm-cache Python-skill requests.
Problem + constraints
Unstructured job postings are expensive to clean and slow to query directly. The system needed to deduplicate reposts, normalize inconsistent skill language, move blocking Spark work out of the request path, and expose aggregation results at interactive API latency.
- The input contains unstructured text, inconsistent terminology, and duplicate reposts.
- Distributed joins can produce partition memory skew when high-cardinality keys are unevenly distributed.
- Multi-attribute aggregation queries need predictable latency without recomputing the same results.
- The processing, API, cache, and database services need repeatable local and CI environments.
- Public results use audited aggregates without exposing raw descriptions or local database credentials.
Contribution boundary
- Built an asynchronous Python and PySpark service that reconciled a 500,000-record real-posting sample into 411,135 canonical postings, recorded 88,865 heuristic duplicate candidates, and produced 156,491 matches across 73 canonical skills.
- Developed FastAPI endpoints with a Redis cache-aside layer; a populated development-scale benchmark measured 0.738ms p95 over 1,000 sequential warm Python-skill requests with 1,000 of 1,000 cache hits.
- Instrumented partition skew and repartitioned on a high-cardinality composite join key; the validated optimized local run completed in 585.748 seconds and recorded skew ratios of 1.057 before and 1.008 after.
- Containerized the services with Docker, designed PostgreSQL composite-key indexes, and implemented unit, Spark contract, service-integration, and image-build checks in GitHub Actions.
- Production adoption, uptime, or contractual service-level guarantees.
- Representation of the entire labor market or causal conclusions about hiring trends.
- Benchmark behavior on infrastructure or workloads outside the measured project environment.
System design, explained
The 30-second explanation
Think of this as organizing a messy stack of job advertisements into a searchable skills index. A background worker cleans repeated postings and links different phrases to the same skill. The API answers filtered questions from prepared data. A separate Next.js dashboard lets visitors explore an audited aggregate snapshot with charts and exports.
Implementation statusValidated locally on a 500,000-record real-posting sample. Warm-cache timings are development measurements, not production guarantees. AWS execution adapters are implemented but deployment is not verified.
Follow the data
A map of the moving parts.
Select any stage to inspect its input, work, output and failure behavior.
Example, not a live resultDashboard example: select Python and a month range to see matching postings and their share of retained records. Counts describe this sample, not all hiring activity.
01 / Submit background work
The API accepts a run and returns 202. The client can poll its status without waiting for Spark.
Worker → Spark processing in lane 02; PostgreSQL remains the durable run-status record
02 / Build the searchable index
This batch path is separate from an analytics read request.
Committed postings + skills → query tables; invalidate demand cache after commit
03 / Serve a filtered question
Cache-aside: check the fast copy first, read the database only when needed.
Redis lookup · two possible routes
JSON aggregation response → API consumer
04 / Explore verified results
The public dashboard reads aggregates from an audited run, not the local processing API.
Snapshot → browser filters, charts and aggregate downloads
Technology, with a purpose
What each tool is responsible for.
Languages, frameworks and services play different roles. Here is why each belongs in this system.
| Technology | Responsibility | Why this choice |
|---|---|---|
| Next.js + ReactPublic frontend | Renders verified aggregates as interactive charts and tables. | Deploys on existing Vercel hosting without exposing the processing API. |
| TypeScript + CSS modulesUI implementation | Defines snapshot types, filters, responsive layouts and exports. | Keeps the display contract explicit and the page lightweight. |
| PythonLanguage + worker | Coordinates asynchronous jobs and publication. | Keeps processing orchestration separate from API request latency. |
| PySparkBatch processing | Normalizes, deduplicates, repartitions and extracts skills. | Expresses distributed joins and transformations, with extra runtime and memory complexity. |
| FastAPIAPI framework | Accepts runs, exposes status and serves filtered aggregations. | Provides explicit validated boundaries for both background work and reads. |
| PostgreSQLRelational database | Stores durable run state, postings, skills and relationships. | Transactions, upserts and indexes support repeatable publication and filtered queries. |
| RedisQueue + cache | Carries work messages and caches demand-query responses. | Decouples workers and accelerates repeated reads; durable state stays in PostgreSQL. |
| DockerRuntime packaging | Runs separate API and Java/PySpark worker images. | Makes the service topology repeatable locally and in CI. |
| GitHub ActionsContinuous integration | Runs unit, Spark contract, integration and image-build checks. | Checks the boundaries together when code changes. |
| EMR Serverless adapterOptional AWS execution | Submits and polls managed Spark jobs. | Allows a cloud execution path; the implemented adapter is not a verified deployment. |
Interview walkthrough
How I explain the design.
Start with the problem, follow one input, then explain the tradeoff.
- 01
The public Next.js dashboard loads an audited JSON snapshot. It filters aggregates in the browser; it does not call an exposed job-submission endpoint.
- 02
I separate the write-heavy batch system from the read-heavy API. A run gets a durable ID, enters Redis, and is processed by a worker.
- 03
PySpark cleans postings, detects reposts and maps aliases to skills. The publisher commits normalized tables before invalidating cached results.
- 04
For a query, FastAPI checks Redis. A hit returns immediately; a miss queries indexed PostgreSQL tables and caches the response.
- 05
I can explain the frozen real-posting sample and warm-cache measurements, but I would need repeated larger runs and production telemetry before claiming scale or a service guarantee.
Decisions + tradeoffs
Distribute cleaning and extraction with PySpark
Python coordinates asynchronous processing while PySpark handles cleaning, deduplication, and normalized skill extraction across the posting corpus.
Repartition on a composite join key
Spark workers are repartitioned around the high-cardinality key used by repost comparison, while each run records before-and-after partition profiles.
Cache aggregation responses in Redis
FastAPI uses a cache-aside path for repeated multi-attribute aggregation queries.
Index the PostgreSQL query shape
Composite-key indexes are designed around the filters and aggregations exposed by the REST API.
Evidence ledger
Claims are labeled by what a reader can inspect now. Withheld items include the measurement needed before publication.
Real-posting processing
Verified500,000 input records produced 411,135 canonical postings, 88,865 heuristic duplicate candidates, and 156,491 skill-match rows across 73 canonical skills.
- Method
- Ran the asynchronous API-to-worker-to-Spark-to-PostgreSQL path and independently reconciled Parquet output counts with PostgreSQL.
- Source / path
- docs/BENCHMARK_2026-10-29.md · October 29, 2026
Warm-cache latency
VerifiedA populated local benchmark returned 1,000 of 1,000 Redis cache hits at 0.574ms p50, 0.738ms p95, and 1.86ms maximum latency.
- Method
- Sent 1,000 sequential warm Python-skill requests against populated PostgreSQL tables through the FastAPI metrics endpoint.
- Source / path
- docs/BENCHMARK_2026-10-29.md · October 29, 2026
Scale and speedup claims
Withheld18% duplicate removal, 500+ matched skills, and a 40-to-12-minute Spark reduction are not claimed. The measured run processed 500,000 real records.
- Method
- Require an immutable production-scale dataset plus repeated same-input baseline and optimized runs before publishing these figures.
- Source / path
- Explicit validation boundary in docs/BENCHMARK_2026-10-29.md
Production outcomes
WithheldProduction adoption, uptime, operating cost, and long-term service levels are not claimed.
- Method
- Publish production telemetry and a dated benchmark protocol before adding those claims.
- Source / path
- Outside current project evidence
Not production-ready
- The frozen real-posting sample contains 500,000 real records and does not demonstrate production-scale throughput.
- Only one optimized 585.748-second local Spark run is recorded; there is no same-input baseline series.
- The 0.738ms p95 result is a populated development-scale warm-cache benchmark, not a production SLA.
- No production adoption, uptime, or cost history is presented.
- Extend the pinned 500K sample with representative sampling and audited duplicate labels.
- Run repeated same-input baseline and optimized Spark trials with cluster, partition, and memory settings recorded.
- Monitor Spark skew, cache hit rate, API latency, database plans, and data freshness in a deployed environment.
- Introduce durable job state, retries, and dead-letter handling for long-running asynchronous work.