Databricks Senior Data Engineer -Pyspark Location NY onsite About the Role The Senior Data Engineer Consultant will implement and optimize enterprise-grade Databricks data platforms across AWS and GCP. You'll collaborate with architecture leads on design/standards and deliver high-quality pipeline implementations with exceptional technical communication. Outstanding verbal and written communication is required—you'll partner with leads, document implementations, and provide knowledge transfer to client teams. Key Responsibilities ● Implement scalable Databricks pipelines using Spark/PySpark across AWS and GCP. ● Design and build optimized ETL/ELT integrating diverse sources into Delta Lake. ● Implement Spark job tuning, cluster optimization, and storage partitioning. ● Build monitoring/alerting/observability for production reliability. ● Author complex BigQuery SQL for cross-platform analytics. ● Collaborate with architecture leads on design reviews, standards adherence, and technical implementation. ● Create comprehensive documentation, runbooks, and implementation guides. ● Participate in code reviews and provide hands-on technical guidance. ● Develop automated testing and data validation frameworks. Required Qualifications ● Bachelor's/Master's in CS, Data Engineering, or related field. ● 7+ yrs data engineering | 4+ yrs Databricks/Spark hands-on. ● Expertise: Spark, PySpark, Delta Lake, SQL optimization. ● Proven Databricks on AWS (S3/Glue) + GCP (GCS/BigQuery). ● Advanced Python (Pandas/NumPy) + pytest testing. ● Cloud services: AWS S3/Glue, GCP BigQuery/Dataflow. ● Exceptional communication—presentations + technical writing. ● Consulting experience: End-to-end project delivery. ● Git + CI/CD proficiency. Preferred Qualifications ● Real-time streaming (Spark Streaming, Kafka, Delta Live Tables). ● Databricks workspace/cluster/Unity Catalog management. ● Docker/Kubernetes. ● Media/entertainment or enterprise analytics experience. ● Jira/Confluence/Lucidchart Neha Chaudhary Team Lead – Recruitment e: neha.chaudhary@compunnel.com o: (+1) 609-606-9010 ext.2469 m: (+1) 732-743-9949 HQ: 4390 Route 1 North, Suite 302, Princeton, NJ 08540, USA. Survey Link : The survey is anonymous, and the information collected will be analysed in aggregate to support initiatives that improve the total experience across Compunnel. — Related