Big Data Engineer - Lisboa

Detalhes da oferta

responsabilidades chave

• <p>Individuals within the Big Data Engineer role ensure that data pipelines are scalable, repeatable, and secure, and can serve multiple users. They help facilitate getting data from a variety of different sources, getting it in the right formats, assuring that it adheres to data quality standards, and assuring that downstream users can get that data quickly. This role usually functions as a core member of an agile team.</p><p>These professionals are responsible for the infrastructure that provides insight from raw data and handles diverse sources of data seamlessly. They enable big data and batch/real-time analytical solutions that leverage emerging technologies. Additional responsibilities include developing prototypes and proof of concepts for the selected solutions, and implementing complex big data projects with a focus on collecting, parsing, managing, analysing and visualizing large sets of data using multiple platforms. They understand how to apply technologies to solve big data problems and to develop innovative big data solutions.</p><p>The Big Data Engineer is a technical job that requires substantial expertise in a broad range of software development and programming fields. These professionals have knowledge of data analysis, end user requirements analysis, and business requirements analysis to develop a clear understanding of the business needs and to incorporate these needs into technical solutions. They have a solid understanding of physical database design principles, and the system development life cycle. These individuals must work well in a team environment.</p><p>Primary Responsibilities</p><ul><li>Designing, developing, constructing, testing and maintaining the complete data management & processing systems – Data Pipelines</li><li>Aggregate & Transform raw data coming from a variety of data sources to fulfill the functional & non-functional business needs – Data Transformation</li><li>Discovering various opportunities for data acquisitions and exploring new ways of using existing data – Data Ingestion</li><li>Creating of data models to reduce system complexity and hence increase efficiency and reduce cost – Data Architecture & Models</li><li>Performance optimization & monitoring: automating processes, optimizing data delivery & re-designing the complete architecture to improve performance</li><li>Proposing ways to improve data quality, reliability & efficiency of the whole system – Data Quality</li><li>Creating a solution by integrating a variety of programming languages and tools together – Data Value</li><li>As a senior, it is expected to be very well communicative and to be extravert for what concerns solutions (technical-architectural), actively helping and supporting the other team members.</li></ul>

qualificações

• <p>Store:</p><ul><li>Data Modelling</li><li>Data Architecture</li><li>Big Data Framework / Hadoop: HDFS, Squid, Spark, Conda, Yarn and MapReduce</li><li>NoSQL Databases: Cassandra, Hbase, MongoDB</li></ul><p>Access & Transport – Connectivity:</p><ul><li>ETL ( Extract, Transform, Load ) : Informatica</li><li>Big Data Framework / Hadoop : Flume & Sqoop, Yarn, Zookeeper</li></ul><p>Enrich :</p><ul><li>Real-time processing framework – Apache Spark</li><li>Big Data Framework / Hadoop : PIG, Hive</li><li>SQL and NoSQL</li><li>Machine learning (nice to have): Python & algorithms</li></ul><p>Provision :</p><ul><li>Workflow</li><li>Programming : Java, Python (and optionally Scala)</li></ul><p>Development methodologies :</p><ul><li>Agile : Safe or Spotify</li><li>DataOps</li></ul><p>Ancillary capabilities :</p><ul><li>Very strong communication skills</li><li>Problem Solving</li><li>Teamwork</li><li>Innovation</li></ul>