Job Description

Summary

We need someone with 4+ years of experience in web data engineering, API integration, or similar roles, who has strong experience with Python libraries (requests, web scraping frameworks (Scrapy, BeautifulSoup, Selenium), and modern browser automation tools (Playwright, Puppeteer). You should be comfortable designing and maintaining robust data collection pipelines from public APIs and web sources, and have a deep understanding of web technologies like HTML, CSS, JavaScript, REST APIs, and GraphQL. You should have up-to-date knowledge of proxy management, session handling, and TLS techniques. Bonus points if you have experience with Rust and/or Go web scraping frameworks for performance, and familiarity with distributed systems and job queues.

• Design and maintain robust data collection pipelines from public APIs and web sources.
• Build and optimize web scraping systems for publicly accessible content.
• Transform and normalize collected data into structured formats for downstream processing.
• Monitor data quality, collection performance, and compliance with platform guidelines.

• Design and maintain robust data collection pipelines from public APIs and web sources.
• Build and optimize web scraping systems for publicly accessible content.
• Transform and normalize collected data into structured formats for downstream processing.
• Monitor data quality, collection performance, and compliance with platform guidelines.

Requirements :

• Either or both of
  • Strong experience with Python libraries (requests, web scraping frameworks (Scrapy, BeautifulSoup, Selenium), modern browser automation tools (Playwright, Puppeteer)
  • Strong experience with Rust and/or Go web scraping frameworks for performance
• Deep understanding of web technologies: HTML, CSS, JavaScript, REST APIs, GraphQL
• Understanding of network protocols, DNS, and web infrastructure
• Familiarity with data formats and serialization (JSON, Parquet, Protocol Buffers)
• Knowledge of proxy management, session handling, and TLS techniques
• Strong background in data validation, cleaning, and transformation pipelines
• 4+ years of experience in web data engineering, API integration, or similar roles
• Experience working in cross-functional teams with strong communication skills

Bonus requirements :

• BS/MS in Computer Science, Engineering, or equivalent experience
• Experience with distributed systems and job queues (Redis, Celery, Amazon SQS)
• Knowledge of web scraping practices and compliance frameworks
• Familiarity with Kubernetes for scalable data collection
• Experience with database systems for storing and indexing collected data
• Public GitHub repositories demonstrating web scraping or API integration projects

Benefits :

• Medical, Dental, Vision, Life Insurance, STD and LTD Plans
• FSA - Medical and Dependent Care
• EAP and wellness programs
• Productivity stipends
• 13 Paid Holidays
• Unlimited PTO
• Flexible work environment - 100% remote
• Bi-annual company/team meetups
• 401(k) plan with employer matching contributions
• Annual review for salary raises

Job Tags

Similar Jobs