We are seeking a skilled Mid-Level Databricks Engineer to join our data engineering team. The ideal candidate will play a key role in the migration of Hadoop workloads to the Databricks Lakehouse platform, contributing to the development, optimization, and monitoring of scalable and repeatable data solutions. You will work closely with cross-functional teams to ensure successful migration, enablement of advanced Databricks features, and ongoing performance improvements.
Key Responsibilities
Identify and categorize existing Hadoop jobs (ETL, batch, streaming) and data sources to support migration planning.
Participate in selecting scalable and repeatable migration use cases for Minimum Viable Product (MVP) initiatives.
Provision and configure Databricks workspaces, ensuring Lakehouse architecture and federation capabilities are enabled.
Execute pilot migrations of representative Hadoop workloads using tools such as Databricks Migration Accelerator or partner solutions.
Validate and monitor post-migration performance, cost efficiency, and data integrity.
Enable and test advanced Databricks features (e.g., Liquid Clustering, Lakehouse AI monitoring, Serverless warehouse, Unity Catalog) to optimize data layout and pipeline health.
Engage with data engineering, analytics, and governance teams to gather feedback and document learnings, blockers, and feature gaps.
Track and report on key success metrics such as migration time, query latency, cost savings, and feature adoption.
Contribute to the development of a phased roadmap for full-scale migration and advanced feature rollout.
Qualifications
Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent professional experience.
5-8 years of experience in data engineering, with hands-on experience in Databricks and Hadoop ecosystems.
Proficiency in Spark (Scala or PySpark), ETL pipeline development, and data migration practices.
Experience with Lakehouse architecture, Delta Lake, and Databricks advanced features (e.g., Lakehouse Federation, Liquid Clustering, Unity Catalog) is a strong plus.
Solid understanding of cloud platforms (Azure) and data governance concepts.
Strong analytical and problem-solving skills with attention to detail.
Ability to work collaboratively in cross-functional teams and communicate effectively with both technical and non-technical stakeholders.
Preferred Skills
Experience with Databricks Migration Accelerator or similar migration tools.
Familiarity with synthetic data generation and pipeline health monitoring.
Understanding of performance tuning, query optimization, and cost management on cloud data platforms.
Ability to document migration patterns and contribute to best practices for broader adoption.
What We Offer
Opportunity to work with cutting-edge data technologies and drive impactful migration projects.
...Job Description Private Equity Fund seeking Accounting Associate Our Radnor, PA Private Equity firm is client is seeking a Private Equity Fund Accountant to join their organization. In office position. The Private Equity Fund Accounting Associate will...
...Description Role Overview: We are seeking a seasoned AWS DevOps Engineer to join our growing AI Platform team, responsible for building... ...within the AI environment. Partner with the IT security and cloud teams to ensure adherence to cybersecurity insurance and data...
...Clinical Research Coordinator Houston, TX (Local Candidates Only) We are currently seeking experienced and motivated Clinical Research Coordinators (CRCs) to join our team in Houston, TX. This role involves managing and coordinating clinical trials at physician practices...
...licensing departments, requiring in-person and online document drop-off and pick-up, and... .... Input, cross-reference, and retrieve data from multiple internal and external databases... ...work experience in an administrative or data entry capacity. ~ High School Diploma, GED, or...
...a course basis each term. Salary Range for Adjuncts teaching in this academic area is $500.00 - $3,750.00 per course. The Theatre and Performance Studies (TAPS) program Production Manager is responsible for the performance of a variety of managerial, supervisory...