Job Summary
Design and deploy scalable data solutions by building data pipelines, integrating data silos, and developing backend APIs. Collaborate across teams in an Agile environment to deliver reliable, data-driven products.
Responsibilities
- Design, develop, and deploy data tables, views, and marts in data warehouses, operational data stores, data lakes, and data virtualization platforms to support business analytics.
- Extract, clean, transform, and manage data flows, including web scraping, to ensure accurate and consistent data availability.
- Build, launch, and maintain efficient large-scale batch and real-time data pipelines using data processing frameworks to support production environments.
- Integrate and consolidate data silos in scalable and compliant ways to enable unified data access.
- Collaborate with Project Managers, Data Architects, Business Analysts, Frontend Developers, Designers, and Data Analysts to build scalable, data-driven products.
- Develop backend APIs and manage databases to support application functionality and performance.
- Participate actively in Agile development processes, including Continuous Integration and Continuous Delivery practices.
- Engage in pair programming and code reviews to maintain high-quality code standards and knowledge sharing.
Preferred competencies and qualifications
- Proficient in data cleaning and transformation using tools such as SQL, pandas, or R to ensure data accuracy and consistency.
- Skilled in building ETL pipelines using technologies like SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container tasks, EventBridge, AWS Glue, and Spring Framework.
- Experienced in database design and working with various databases including SQL, PostgreSQL, AWS S3, Athena, MongoDB, PostGIS, MySQL, SQLite, VoltDB, and Cassandra.
- Familiar with cloud platforms such as AWS, Azure, and Google Cloud Platform (GCP) for big data engineering.
- Knowledgeable about system design, data structures, and algorithms to optimize data processing.
- Familiar with data modeling, data access, and storage infrastructures like Data Mart, Data Lake, Data Virtualization, and Data Warehouse.
- Experienced with REST APIs and web protocols to support backend integration.
- Familiar with big data frameworks and tools such as Hadoop, Spark, Kafka, and RabbitMQ.
- Skilled in web scraping using tools like BeautifulSoup, CasperJS, PhantomJS, Selenium, and Node.js.
- Understand data governance policies, access control, and security best practices to ensure data protection.
- Comfortable scripting in languages such as SQL and Python.
- Experienced working in both Windows and Linux development environments.
- Interested in bridging engineering and analytics teams to enhance data utilization.
You apply on MyCareersFuture, where this job was posted. Peter Nowak isn’t the employer and doesn’t check listings.