Apache Flink Engineer

PeoplePlus Tech

₱70-130K[Monthly]

Remote5-10 Yrs ExpBachelorFull-time

Remote Details

Open Country：Philippines

Language Requirements：English

This remote job is open to candidates in specific countries. Please confirm if you want to continue despite potential location restrictions

Job Description

Kafka Integration - Setting up Flink jobs to seamlessly connect to specific Kafka topics, defining consumer groups, and managing data serialization/deserialization.
Stream Processing Logic - Writing Flink data processing logic using DataStream API to filter, aggregate, window, and manipulate data streams received from Kafka.
State Management - Implementing stateful operations within Flink jobs to maintain context across events within a data stream, such as tracking running totals or session information.
Flink Job Optimization - Tuning Flink job parameters for performance, parallelism, and fault tolerance to handle high-volume data streams efficiently.
Monitoring and Alerting - Setting up monitoring systems to track Flink job health, identify potential issues, and trigger alerts for operational concerns.
Developing Flink applications - Writing code to define data pipelines using Flink's DataStream API, including data ingestion, transformation, and output operations.
Stream processing design - Architecting real-time data processing solutions using Flink's capabilities for state management, windowing, and fault tolerance.
Performance optimization - Monitoring and tuning Flink jobs to ensure efficient data processing and high throughput.
Integration with other systems - Connecting Flink applications to data sources like Kafka, databases, and other data processing systems.
Cluster management - Deploying and managing Flink jobs on distributed computing clusters like Kubernetes.

Thorough understanding of Kafka architecture, producers, consumers, partitions, brokers, and topic management.
Knowledge of Kafka architecture, producers, consumers, topic management, and Kafka Streams API.
Familiarity with Java development to write Flink applications and interact with Kafka using Java libraries.
Understanding of stream processing principles, event-time processing, windowing techniques, and exactly-once semantics.
Deep understanding of Flink concepts like DataStream API, operators, state management, windowing, and fault tolerance mechanisms
Familiarity with distributed computing principles and concepts like fault tolerance, load balancing, and data partitioning
Experience with other data processing tools like Kafka, Spark, or Hadoop can be beneficial
Understanding SQL can be valuable for working with Flink's SQL API
Processing live data from sensors or applications to generate near-instantaneous insights.
Analyzing financial transactions in real-time to identify suspicious activity
Performing live log aggregation and analysis to detect system issues
Tracking stock levels in real-time to optimize inventory management

apacheflink