Joao is available for hire

Joao Gabriel Oliveira

Verified Expert in Engineering

Data Engineer and Developer

Location

Lisbon, Portugal

Toptal Member Since

December 23, 2022

Joao is an experienced, challenge-driven, 专注于数据工程和机器学习的注重细节的软件架构师. 他在设计和实现结合高效处理的端到端数据解决方案方面非常熟练, 复杂的查询功能, and insights extraction. 具有扎实的计算机科学和数学基础，良好的沟通和团队合作能力, Joao可以为最具挑战性的数据驱动项目提供价值.

Data Engineering 自然语言处理(NLP)Machine Learning Big Data Architecture Data Warehousing Elasticsearch Python SQL Databases NoSQL Database Architecture Database Design ETL Data Pipelines PySpark Full-Stack Python

Portfolio

Izea

Apache Spark, Elasticsearch, Python, Amazon SageMaker...

Izea

Apache Spark, Elasticsearch, Python, Amazon SageMaker，数据建模...

TapInfluence

Scala, Spark, Java, Spring, AWS云开发，SQL, Terraform, PostgreSQL...

Experience

Elasticsearch - 10 years Java - 10 years Python - 10 years Data Modeling - 10 years Data Lakes - 5 years Apache Spark - 5 years Amazon SageMaker - 4 years

Availability

Part-time

Preferred Environment

Python, Amazon Web Services (AWS)， Visual Studio Code (VS Code)， Slack, Jira, GitHub

The most amazing...

...我所做的贡献是建模和协调一个社交媒体数据平台的发展，该平台拥有超过1000万个个人资料和1.5 billion posts.

Work Experience

Data Manager

2022 - PRESENT

Izea

通过过程改进，将平均团队速度提高到50%, better planning, 以及知识共享计划.
通过提供公司社交数据平台的技术方向，支持多个团队.
领导和管理一个五人的团队, 包括后端工程师, data engineers, and data scientists.

Technologies: Apache Spark, Elasticsearch, Python, Amazon SageMaker, Amazon Elastic MapReduce (EMR), Jira, Data Modeling, Data Lakes, Spark Streaming, Amazon Kinesis, AWS Cloud Development, SQL, Data Engineering, PySpark, ETL, PostgreSQL, Data Architecture, Big Data Architecture, Data Warehousing, GPT, 生成预训练变压器(GPT), 自然语言处理(NLP), JavaScript, NoSQL, Amazon Web Services (AWS), Data, Docker, CI/CD Pipelines, DevOps, Data Visualization, Machine Learning, Spark, Programming, Semantics, Data Pipelines, Amazon弹性容器服务(Amazon ECS), ETL Implementation & Design, Amazon S3 (AWS S3), Business Intelligence (BI), APIs, Reporting, Databases, Data Transformation, Database Architecture, Database Design, Data Science, Architecture

高级数据和机器学习工程师

2018 - 2022

Izea

建模和协调一个拥有超过1000万个人资料和1个社交媒体数据平台的部署.5 billion posts, 通过复杂的索引模式设计集成批处理和流处理技术.
率先执行机器学习模型作为核心平台元素, 结合文本特征提取和各种回归分类算法来预测受众人口统计.
在指定数据产品需求时，作为产品和数据团队之间的主要联络点.
通过领导实施流程，支持三个新产品的推出.

Technologies: Apache Spark, Elasticsearch, Python, Amazon SageMaker，数据建模, Data Lakes, Spark Streaming, Amazon Kinesis, AWS Cloud Development, SQL, Data Engineering, PySpark, ETL, PostgreSQL, Data Architecture, Big Data Architecture, Data Warehousing, 生成预训练变压器(GPT), GPT, 自然语言处理(NLP), JavaScript, NoSQL, Amazon Web Services (AWS), Data, Docker, CI/CD Pipelines, DevOps, Data Visualization, Machine Learning, Spark, Programming, Semantics, Data Pipelines, Amazon弹性容器服务(Amazon ECS), ETL Implementation & Design, Amazon S3 (AWS S3), Business Intelligence (BI), APIs, Reporting, Databases, Data Transformation, Database Architecture, Database Design, Data Science, Architecture

Senior Software Architect

2017 - 2018

TapInfluence

协调系统向微服务架构的转变，重点关注分析和搜索组件.
为实现产品搜索功能上的相关性评分改进做出了重大努力.
在关键技术和架构决策方面为工程副总裁提供卓越的协助.

Technologies: Scala, Spark, Java, Spring, AWS云开发，SQL, Terraform, PostgreSQL, NoSQL, Amazon Web Services (AWS), Data, Docker, CI/CD Pipelines, DevOps, Data Visualization, Programming, Data Pipelines, Amazon弹性容器服务(Amazon ECS), ETL Implementation & 设计，Amazon S3 (AWS S3)， api，数据库，数据库架构，数据库设计，架构

高级软件架构师|合伙人

2010 - 2017

Amtera语义技术

使用最先进的语义搜索和自然语言处理研究实现不同的产品概念和组件.
为电信行业的各种重要客户建立健全有效的内部解决方案, oil and gas, and IT security industries.
与其他合作伙伴合作实施公司整体战略.

Technologies: Python, MongoDB, Elasticsearch, Linked Data, Semantics, 生成预训练变压器(GPT), GPT, 自然语言处理(NLP), Data Modeling, SQL, Scala, Data Engineering, ETL, PostgreSQL, Data Architecture, Big Data Architecture, NoSQL, Data, Data Visualization, Machine Learning, Spark, Programming, ETL Implementation & 设计，api，数据库，数据库架构，数据库设计，数据科学，架构

Experience

Izea核心社交媒体数据平台

使用数据湖架构进行摄取的基于云的数据平台, process, index, 并查询社交媒体数据, 涵盖全文搜索和分析用例.

我是关键架构师之一，在实现该平台的过程中也发挥了核心作用.

Izea基于nlp的机器学习模型

一套机器学习模型，使用词嵌入管道和其他监督和非监督方法来预测社交媒体简介的受众人口统计数据.

我是这个项目的主要架构师和开发者.

Skills

Languages

Python, SQL, Java, Scala, JavaScript

Frameworks

Apache Spark, Spark, Spring

Libraries/APIs

Spark Streaming, PySpark, Node.js, PyTorch

Tools

GitHub, Amazon Elastic MapReduce (EMR), Amazon SageMaker, Slack, Jira, Amazon弹性容器服务(Amazon ECS), Terraform

Paradigms

ETL, ETL Implementation & 设计、数据库设计、数据科学、DevOps、商业智能(BI)

Platforms

Amazon Web Services (AWS)、Docker、Visual Studio Code (VS Code)

Storage

Elasticsearch, Databases, NoSQL, Data Pipelines, Amazon S3 (AWS S3), Database Architecture, Data Lakes, MongoDB, PostgreSQL

Other

Data Modeling, Algorithms, Programming, Time Complexity Analysis, Space Complexity Analysis, Linked Data, Semantics, 自然语言处理(NLP), Machine Learning, Data Engineering, Data Architecture, Big Data Architecture, Data Warehousing, Data, APIs, Data Transformation, Architecture, GPT, 生成预训练变压器(GPT), Amazon Kinesis, AWS Cloud Development, Compilers, Number Theory, Deep Learning, CI/CD Pipelines, Data Visualization, Reporting

Education

2006 - 2012

计算机科学学士学位

里约热内卢联邦大学| UFRJ -里约热内卢，巴西

Certifications

AUGUST 2022 - PRESENT

机器学习与Python:从线性模型到深度学习

MITx Online

MAY 2013 - PRESENT

MongoDB for Developers

MongoDB University

Collaboration That Works

How to Work with Toptal

在数小时内，而不是数周或数月，我们的网络将为您直接匹配全球行业专家.

Share your needs

在与Toptal领域专家的电话中讨论您的需求并细化您的范围.

Choose your talent

在24小时内获得专业匹配人才的简短列表，以进行审查，面试和选择.

开始你的无风险人才试验

与你选择的人才一起工作，试用最多两周. 只有当你决定雇佣他们时才付钱.

对顶尖人才的需求很大.

Start hiring