Job Description
Roles & Responsibilities :
We are seeking a skilled Senior Data Engineer with expertise in Databricks to join our dynamic data team. The ideal candidate will design, build, and maintain scalable data pipelines and architectures to support our organization's data-driven initiatives and leverage Databricks to process large-scale datasets, optimize data workflows, enable advanced analytics and machine learning, and integrate Power BI for data visualization and reporting.
Key Responsibilities
5-8 years of professional work experience in a relevant field
Proficient in Microsoft Fabric platform, Azure Databricks, ADF, Delta Lake, SQL Data Warehouse, Unity Catalog.
Good Experience on Microsoft Dynamics 365Experience/ prior knowledge on semi structure data and Structured Streaming, Azure synapse, data lake, data warehouse.
Proficient in creating Azure Data Factory pipelines for ETL/ELT processing; copy activity, custom Azure development etc.
Good knowledge of SQL and Python for data manipulation, transformation, and analysis
Understand business requirements to set functional specifications for reporting applications
Data Pipeline Development: Design, develop, and maintain robust, scalable data pipelines using Databricks, Apache Spark, and other cloud-based technologies.
Data Integration: Ingest, transform, and integrate data from diverse sources, including APIs, databases, streaming platforms, and third-party systems, into Databricks for analytics and reporting.
Power BI Integration: Develop and optimize data models and datasets in Databricks for use in Power BI, ensuring efficient data connections and high-quality visualizations.
Performance Optimization: Optimize data workflows, API calls, and queries on Databricks and Power BI for performance, cost-efficiency, and scalability.
Data Modeling: Build and maintain data models to support business requirements, ensuring data quality, consistency, and accessibility for analytics and reporting.
Cloud Integration: Implement data solutions on cloud platforms (e.g., AWS, Azure, GCP) integrated with Databricks, Power BI, and API ecosystems.
Security & Compliance: Implement data governance, security, and compliance best practices within Databricks, Power BI, and API environments.
Technical Skills:
Proficiency in Databricks, including Delta Lake, Spark SQL.
Strong programming skills in Python, Scala, or Java.
Experience with Apache Spark for big data processing.
Knowledge of SQL for querying and transforming data.
Proficiency in Power BI for creating data models, DAX queries, and interactive dashboards.
Preferred Qualifications
Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).
Experience with real-time data processing and streaming (e.g., Kafka, Spark Streaming).
Additional Information
Educational qualification:
BE/B.Tech or Equivalent Degree
Experience :
2-5 Years
Mandatory/requires Skills :
Strong hands-on expertise in Python (or Java), NLP, RegEx, SpaCy, NLTK and DevOps
Preferred Skills :
