Data Engineer (Lakehouse/BigData/Data Platform)

Overview

Viettel Cyber Security (VCS) is a leading cyber security firm, dedicated to protecting digital infrastructures worldwide while conducting in depth research and analysis for security solutions. Leveraging 100% in house development, VCS offers a comprehensive range of cutting-edge cyber security solutions, protecting governments, financial institutions, large corporations and SMEs against sophisticated cyber threats in the digital age.

VCS is internationally recognized for its Champion titles at Pwn2Own 2023 & 2024, consistent Top 5 global rankings, the discovery of400+ zero-day vulnerabilities, and multiple prestigious honors, including Gold – Best Cyber Security Company in Asia.

Consolidated Platform is a unified platform that consolidates and connects data from multiple cybersecurity, information security monitoring, and IT system management products within an organization into a single, integrated view. The platform’s core strength lies in its ability to aggregate data and establish cross-source relationships across previously siloed systems. This enables cybersecurity monitoring and management professionals, as well as C-suite executives, to gain both a high-level overview and detailed insights for investigation without having to work across multiple tools. We are looking for two Experienced-level and one Senior-level candidate to join the project; the requirements and job descriptions for each level are outlined below.

Responsibilities

Job Description:

As an Experienced Data Engineer, you will take end-to-end ownership of assigned data pipelines — from data ingestion through to application consumption — with responsibility for testing and basic data quality control.

1. Build and Scale Data Pipelines

• Develop ingestion jobs to bring data from new sources, including REST APIs, XML/CSV/JSON files, and Kafka topics, into the lakehouse; handle incremental loading, checkpoints, and schema changes over time.
• Standardize, clean, and deduplicate data into a common schema; develop matching and data consolidation logic across sources, resolving conflicts when different sources describe the same entity in different ways.
• Build the serving layer for applications and reporting, including upserts to PostgreSQL/MongoDB, ensuring data is delivered on time and remains consistent with the source snapshots.
• Develop and maintain orchestration DAGs, including dependencies, retries, backfills, and scheduling.

2. Data Quality and Operations

• Define data expectations for assigned tables, including schema, constraints, volume, and freshness, and implement automated tests at layer boundaries; investigate and resolve data quality alerts.
• Monitor jobs, debug failures, and troubleshoot common performance issues such as data skew, small files, and OOM (Out of Memory); optimize queries and table structures.
• Write unit tests for transformation logic and follow the team’s branching/commit conventions, code review practices, and CI processes.

3. Collaboration

• Work closely with Product, BI, and AI teams to understand data consumption requirements; maintain data documentation including metadata, lineage, and data dictionaries so that downstream systems and AI Agents can use data with the correct context and access permissions.

——————-

As a Senior Data Engineer, you will design the data platform so that the team can onboard new data sources quickly without compromising quality. You will establish technical standards for pipelines, data models, and the semantic layer supporting analytics and AI Agents, as well as build data quality measurement frameworks to support release decisions.

You will be responsible for the reliability, performance, and security of the platform handling sensitive cybersecurity data, while providing technical leadership and mentoring to the team.

1. Architecture and Technical Standards

• Design an end-to-end lakehouse architecture for multiple tenants, including data layering, partitioning and table organization strategies, and data contracts between layers.
• Standardize the pipeline development framework, including job templates, resource profiles, flow registration mechanisms, and checkpoints, so that onboarding a new data source becomes a repeatable process rather than requiring a new design from scratch.
• Design data models and semantic layers, including consistent definitions of metrics, entities, and lineage, to support reporting, analytics, and AI Agent data queries.

2. Data Quality, Reliability, and Performance

• Build a data quality measurement framework that runs across the full dataset on every pipeline run, rather than relying only on sample-based validation; define metrics, thresholds, quality gates, alerts, and operational runbooks.
• Size resources based on actual measurements such as data volume, skew, and cardinality, rather than trial and error; address large-scale performance challenges including data skew, small files, compaction, snapshot expiration, and storage costs.
• Establish data governance, including data catalogs, lineage, sensitive data classification, and role-based access control.
• Integrate CI/CD, automated testing, and code review standards into the pipeline development process.

3. Technical Leadership and Collaboration

• Mentor Data Engineers on the team, conduct code and design reviews, and establish clear ownership across data domains.
• Evaluate and select technologies for the data platform architecture; collaborate with Product, AI, BI, and Infrastructure teams to bring data into the product.

Qualifications

Experienced Data Engineer

• 3+ years of experience in Data Engineering, with hands-on experience building and operating data pipelines in production.
• Strong proficiency in advanced SQL, including window functions, CTEs, and reading EXPLAIN plans for query optimization; proficient in Python, with experience writing structured code, reusable modules, exception handling, logging, and tests using pytest.
• Hands-on experience with distributed data processing using Spark, preferably PySpark; understand partitioning, shuffle, and join strategies, and be able to use Spark UI to identify performance bottlenecks.
• Experience with an orchestration tool, preferably Airflow or alternatively Dagster/Prefect; able to design DAGs, dependencies, retries, and backfills.
• Understanding of data warehouse/lakehouse architecture, including data layering, data modeling, incremental loading, and schema evolution.
• Experience working with both SQL and NoSQL databases such as PostgreSQL and MongoDB, S3-compatible object storage such as MinIO/S3, and Docker.
• Strong awareness of data quality, including the ability to proactively validate null values, duplicates, and data inconsistencies before delivering datasets to downstream systems.
• Bachelor’s degree in Information Technology, Information Security, or related IT fields.

Bonus Points

• Experience with lakehouse table formats such as Apache Iceberg (preferred), Delta Lake, or Hudi, including time travel, schema evolution, and table maintenance.
• Experience with distributed query engines such as Trino/Presto.
• Experience with data quality frameworks such as Great Expectations, Soda, or dbt Test, and data catalog/lineage tools such as OpenMetadata, DataHub, or Amundsen.
• Experience ingesting near-real-time data through Kafka; stream processing experience is a plus.
• Domain knowledge in cybersecurity, including CVE/CVSS, SIEM/EDR logs, IT asset management, and vulnerability scanning results.
• Experience with Kubernetes or cloud platforms such as AWS, GCP, or Azure.

————–

Senior Data Engineer

• 5+ years of experience in data/software engineering, including at least 2 years of experience designing and operating large-scale data platforms in production.
• Strong proficiency in Spark optimization, including reading physical execution plans and handling data skew, partitioning, shuffle, and file-size optimization.
• Proven experience designing data warehouse/lakehouse architectures, including data layering, data modeling, CDC/incremental processing, versioning, and schema evolution.
• Experience operating orchestration systems across multiple pipelines, including DAG standardization, data SLAs, backfills, and cross-pipeline dependency management.
• Experience building data quality and observability systems for data pipelines, including metrics, alerts, and lineage.
• Strong understanding of system architecture, including SQL/NoSQL, containers/Kubernetes, message queues, and object storage; strong software engineering practices including testing, CI/CD, and code review.
• Experience mentoring engineers and providing technical direction to the team.
• Bachelor’s degree in Information Technology, Information Security, or related IT fields.

Bonus Points

• • Production experience with Apache Iceberg, including compaction, snapshot expiration, and table migration; experience tuning Trino/Presto.
  • Experience designing a semantic layer/data model for AI Agent data querying, including text-to-SQL and MCP standards.
  • Experience implementing data governance frameworks, data contracts, and data ownership models.
  • Experience designing multi-tenant data systems and handling sensitive data in compliance with security and regulatory requirements.
  • Experience with production-scale real-time data processing, including Kafka and stream processing technologies.
  • General knowledge of the cybersecurity domain

Benefits & perks

• Competitive annual total income package
• Annual salary review in March.

• Premium health and personal accident insurance
• Annual health check-up
• Special support for female employees during maternity leave.

• Awards recognizing performance and timely outstanding contributions
• 100% sponsorship for in-depth international professional certification based on position (in case of unsuccessful attempt, don’t worry, VCS will support 50% of the exam fee), special bonuses for employees achieving international professional certifications
• Continuous investment in the latest and most up-to-date learning materials in the field of Information Security from leading global vendors and institutes
• 100% of employees are provided with an Udemy eLearning account – learn anytime, anywhere
• Free participation in professional knowledge sharing programs and seminars from leading domestic and international speakers

• Opportunity to work with diverse domestic and international clients
• Working in a Grade A office – Landmark 72 building with a green space complex and a private area for gym and entertainment (billiards, PES, café, reading, Pingpong)
• 30-minute relaxation daily with Happy hours (4:00 PM – 4:30 PM)
• Sports activities: Swimming, Pickleball, Billiards, Poker, PES, etc.

• Twelve (12) days of annual leave according to Labor Law, three (3) days of company vacation and one (1) traditional holiday on 22/12.

• Personal birthday celebrations: Gifts and birthday cake from the Company
• Gift and participation in company events: Quarterly team building activities, monthly birthday celebrations, Year End Party, vacations, International Women’s Day (March 8th), Vietnamese Women’s Day (October 20th) (female employees receive ½ day off), Viettel Group birthday on June 1st, etc
• Care parents and children of employees on special occasions.