Connecting...

Big Data Engineer

Job title: Big Data Engineer
Contract type: Permanent
Location: Kuala Lumpur
Sector:
Salary: RM7,000-RM10,000 per month
Contact name: Izzah Hayati Saibudin
Contact email: izzah.saibudin@ellwoodconsulting.com.my
Published: 4 days ago

Job description

​About our client

Our client is a multinational regional financial services provider dedicated to providing complete solutions to customers through differentiated segment offerings and an ecosystem that supports simple, fast, and seamless customer experiences, backed by a cohesive and inspired workforce and stakeholder relationships.

Our client is ranked among Malaysia's top banks and has a strong presence in ASEAN, has strong market leadership in Malaysia across targeted products and sectors. In the ASEAN area, the bank has a presence in nine nations.

About the role

You are expected to be responsible for the end-to-end test cycle and work collaboratively with multidisciplinary team members. You will be an integral member of a very dynamic and growing organization and inclusive workplace, tap into your potential for diversified solution portfolio and work with clients in different verticals and peers across the globe.

Your responsibilities

Gather and process raw data at scale.

  • Design and develop data applications using selected tools and frameworks.

  • Read, extract, transform, stage, and load data to selected tools and frameworks.

  • Simplify access to real-time data for internal stakeholders using real-time.

  • Design and implement the banks’ real-time data pipelines.

  • Collaborate with data stakeholders and stewards on the verification and accuracy of the information collected.

  • Provide technical lead to Data Owners and Stewards on data definition, data lineage changes supporting intake process, performing impact analysis, and conducting domain-specific profiling.

  • Monitoring data performance and modifying infrastructure.

  • Custom integration with various banking systems, data warehouses, and analytics systems.

  • Setup of data-access or visualization tools for data scientists such as data science workbench.

  • Develop and implement analytics use cases to yield business value from data and insight.

  • Extracts and transforms structured and unstructured big and small data to generate features, derive impactful insight, visualize information, and support impactful decision-making.

  • Design of data pipeline and implement data ingestion from end-to-end to Big Data repositories.

  • Data Management & Data Governance for Big Data Platform.

You will have

Required Skills:

  • Demonstrated ability to build high-volume data ingestion and streaming pipelines (e.g. AWS Kinesis, Spark Streaming).

  • Excellent knowledge of Python, Scala or Java, and SQL.

  • Big Data Technology and Programming skills and knowledge such as Hive, Hue, Spark, Pyspark, Yarn, HBase, Cloudera, Nifi, Mapreduce, and Linux.

  • Knowledge of advanced data technologies like Kafka, H2O, and ElasticSearch would be an added advantage.

Required Experience:

  • Degree/Master in IT, Computer Science, or related discipline.

  • Minimum of 5 years of relevant industry experience.

  • Working experience with big data technologies (e.g. Spark, Kafka, Hive).

  • Experience designing, building, and scaling a production-ready event streaming system.

  • Experience in optimizing SQL queries (e.g. data partitioning, bucketing, indexing).

  • Experience in building data lake/warehouse solutions consisting of structured and unstructured data.

Get similar jobs like these by email

By submitting your details you agree to our T&Cs