Arquitetura Lambda e Kappa com Spark na AWS: como juntar batch e tempo real sem duplicar pipeline

O Amazon EMR Serverless permite rodar o mesmo Apache Spark tanto para processar o histórico em lote quanto para consumir o stream do Kinesis, o que muda a forma de montar os padrões Lambda e Kappa na prática.

Nota da redação: as atribuições de autoria da arquitetura Lambda a Nathan Marz e do padrão Kappa a Jay Kreps são de conhecimento geral da literatura de engenharia de dados, mas não constavam nas fontes técnicas usadas para verificar este artigo (documentação do EMR Serverless). Não foi possível confirmar essas atribuições especificamente contra uma fonte primária nesta checagem; o núcleo técnico sobre EMR Serverless, Spark e os padrões em si foi verificado normalmente.

O Amazon EMR Serverless permite rodar o mesmo Apache Spark tanto para processar o histórico em lote quanto para consumir o stream do Kinesis, o que muda a forma de montar os padrões Lambda e Kappa na prática.

O Amazon EMR Serverless permite rodar o mesmo Apache Spark tanto para processar o histórico em lote quanto para consumir o stream do Kinesis, o que muda a forma de montar os padrões Lambda e Kappa na prática.

Quem mantém um pipeline de dados na AWS conhece o dilema: o histórico fica num data lake em S3 e é processado em lote, enquanto os eventos que chegam agora passam por um stream no Amazon Kinesis e precisam de resposta em minutos, não em horas. Resolver isso historicamente significava manter duas pipelines separadas, cada uma com seu próprio código, seu próprio time de manutenção e, não raro, resultados que divergem entre si.

É esse o problema que os padrões Lambda e Kappa tentam resolver, e é aí que o Apache Spark rodando sobre o Amazon EMR Serverless entra como peça central da arquitetura. A documentação do EMR Serverless descreve o serviço como uma opção de deployment do Amazon EMR que oferece runtime serverless para frameworks como Spark e Hive, sem que o time precise configurar, otimizar ou operar clusters.

O problema que a arquitetura Lambda resolve (e o preço que cobra)

A arquitetura Lambda, descrita originalmente por Nathan Marz, separa o pipeline em três camadas: a camada batch processa todo o histórico com precisão total, a camada de velocidade processa só os eventos recentes com menor latência e possivelmente menor precisão, e a camada de serving junta as duas visões para quem consulta.

Na prática em AWS isso costuma virar: um job Spark em lote lendo o S3 inteiro e escrevendo em uma tabela otimizada; um job Spark Structured Streaming consumindo o Kinesis em micro-lotes e escrevendo incrementos; e uma camada de consulta (Athena, Redshift Spectrum ou um dashboard) que combina as duas fontes.

O custo dessa arquitetura não é só operacional. É ter a mesma lógica de negócio escrita duas vezes, em dois jobs diferentes, que às vezes divergem silenciosamente quando alguém corrige um bug só na camada batch.

O padrão Kappa: quando dá para eliminar o batch

A alternativa, proposta por Jay Kreps, é o padrão Kappa: existe só a camada de streaming, e o reprocessamento histórico é feito reexecutando o mesmo job de streaming contra um stream reconstruído a partir do início (replay), em vez de manter um job batch paralelo.

Na prática, isso exige que o stream de origem retenha dados por tempo suficiente para cobrir o período que se queira reprocessar, o que impõe uma configuração de retenção maior do que o padrão mais curto usado em streams que só servem consumo em tempo real. Não existe mágica aqui: o Kappa troca a complexidade de manter dois pipelines pela complexidade de garantir que um único pipeline de streaming seja rápido o suficiente para reprocessar meses de dados num tempo aceitável.

Em resumo: Lambda aceita duplicar lógica para ganhar velocidade na camada quente; Kappa aceita um streaming mais pesado para não duplicar nada.

Onde o EMR Serverless muda a equação

O que torna os dois padrões mais viáveis hoje na AWS é poder usar o mesmo motor, Spark, nas duas camadas, sem manter um cluster EMR provisionado rodando 24 horas só para o job batch noturno.

Segundo a documentação, o EMR Serverless calcula automaticamente os recursos que cada job precisa, provisiona workers, executa e desprovisiona quando o job termina, e isso vale tanto para um job batch que roda uma vez por dia quanto para um job de streaming que fica ativo o tempo todo. Alguns conceitos da plataforma importam diretamente para quem desenha esse pipeline:

• Application: a unidade que você cria especificando o release do EMR (que define a versão do Spark) e o tipo de runtime. Times costumam criar aplicações separadas para o job batch e para o job de streaming, com controles de custo independentes.
• Job run: cada submissão roda de forma assíncrona e usa uma runtime role em IAM própria, o que permite dar ao job de streaming acesso de leitura ao Kinesis e ao job batch acesso só ao S3 do data lake.
• Workers: o EMR Serverless escala os workers automaticamente conforme a carga em cada estágio do job, o que no caso do streaming significa absorver picos de volume no Kinesis sem redimensionamento manual.
• Pre-initialized capacity: um pool de workers já aquecidos, configurado pelo parâmetro initial-capacity da aplicação, pensado para jobs sensíveis a tempo de início, relevante justamente para o job de streaming, que não pode esperar minutos até o primeiro worker subir.

Um pipeline concreto: Kinesis, Spark Structured Streaming e S3

Nota da redação: o código desta seção não foi executado em ambiente real. Valide antes de usar em produção.

Para ilustrar como as peças se encaixam, um pipeline Kappa típico começa criando a aplicação Spark no EMR Serverless:

aws emr-serverless create-application \
 --release-label emr-7.1.0 \
 --type "SPARK" \
 --name pipeline-pedidos-streaming \
 --initial-capacity '{"DRIVER":{"workerCount":1,"workerConfiguration":{"cpu":"2vCPU","memory":"4GB"}},"EXECUTOR":{"workerCount":2,"workerConfiguration":{"cpu":"4vCPU","memory":"8GB"}}}'

O job em si, submetido via start-job-run com uma runtime role que tem permissão de kinesis:GetRecords no stream de origem, roda um script PySpark que lê o Kinesis em micro-lotes e grava no S3 em formato Parquet, com checkpoint para garantir exatly-once na escrita:

df = (spark.readStream
 .format("kinesis")
 .option("streamName", "pedidos-stream")
 .option("endpointUrl", "https://kinesis.sa-east-1.amazonaws.com")
 .option("startingPosition", "LATEST")
 .load())

query = (df.writeStream
 .format("parquet")
 .option("path", "s3://data-lake/bronze/pedidos/")
 .option("checkpointLocation", "s3://data-lake/checkpoints/pedidos/")
 .trigger(processingTime="1 minute")
 .start())

Esse mesmo script, com startingPosition ajustado para TRIM_HORIZON (o início da retenção do stream) e apontando para um stream reconstruído, é o que vira o reprocessamento histórico no padrão Kappa, sem escrever um segundo job em lote. A camada de serving fica por conta do Glue Data Catalog registrando a tabela e do Athena ou Redshift Spectrum consultando por cima.

Lambda ou Kappa: qual escolher

A escolha não é estética, é sobre o padrão de acesso aos dados e a tolerância a reprocessamento caro.

Quando o EMR Serverless não é a peça certa

Vale ser direto sobre os limites. O Spark Structured Streaming trabalha em micro-lotes (o trigger do exemplo acima define o intervalo), então a latência mínima prática fica na casa de segundos a poucos minutos, não é a ferramenta certa para resposta em milissegundos, caso em que o Amazon Managed Service for Apache Flink costuma ser mais adequado.

Também não compensa para jobs minúsculos e constantes, onde o overhead de provisionar e desprovisionar workers a cada execução pode custar mais, em tempo de start, do que manter algo sempre ativo. E para quem já tem um cluster EMR provisionado rodando perto de 100% de utilização o dia inteiro, migrar para serverless não traz ganho de custo, o modelo serverless compensa justamente quando há ociosidade para eliminar.

A documentação da AWS não detalha preço nesta página específica, menciona apenas 'straightforward cost controls'. Mas a lógica geralmente associada a runtimes serverless, cobrar pelos recursos efetivamente usados durante a execução do job em vez de manter um cluster provisionado o tempo todo, ajuda a entender por que faz sentido usar o mesmo runtime Spark nas duas camadas da arquitetura Lambda, ou numa única camada na arquitetura Kappa, sem manter infraestrutura ociosa esperando o próximo evento do Kinesis.

Fonte 1: Documentação do Amazon EMR Serverless (https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html)

What is Amazon EMR Serverless? - Amazon EMR

View a markdown version of this page What is Amazon EMR Serverless? - Amazon EMR Documentation Amazon EMR Documentation Amazon EMR Serverless User Guide Concepts What is Amazon EMR Serverless? Amazon EMR Serverless is a deployment option for Amazon EMR that provides a serverless runtime environment. This simplifies the operation of analytics applications that use the latest open-source frameworks, such as Apache Spark and Apache Hive. With EMR Serverless, you don’t have to configure, optimize, secure, or operate clusters to run applications with these frameworks. EMR Serverless helps you avoid over- or under-provisioning resources for your data processing jobs.

EMR Serverless automatically determines the resources that the application needs, gets these resources to process your jobs, and releases the resources when the jobs finish. For use cases where applications need a response within seconds, such as interactive data analysis, you can pre-initialize the resources that the application needs when you create the application. With EMR Serverless, you continue to get the benefits of Amazon EMR, such as open source compatibility, concurrency, and optimized runtime performance for popular frameworks. EMR Serverless is suitable for customers who want ease in operating applications using open source frameworks. It offers quick job startup, automatic capacity management, and straightforward cost controls.

Concepts

In this section, we cover EMR Serverless terms and concepts that appear throughout our EMR Serverless User Guide.

Release version

An Amazon EMR release is a set of open-source applications from the big data ecosystem. Each release includes different big data applications, components, and features that you select for EMR Serverless to deploy and configure so that they can run your applications. When you create an application, specify its release version. Choose the Amazon EMR release version and the open source framework version that you want to use in your application. To learn more about pre-release versions, refer to Amazon EMR Serverless release versions .

Application

With EMR Serverless, you can create one or more EMR Serverless applications that use open source analytics frameworks. To create an application, specify the following attributes:

The Amazon EMR release version for the open source framework version that you want to use. To determine your release version, refer to Amazon EMR Serverless release versions .

The specific runtime that you want your application to use, such as Apache Spark or Apache Hive.

After you create an application, submit data-processing jobs or interactive requests to your application.

Each EMR Serverless application runs on a secure Amazon Virtual Private Cloud (VPC) strictly apart from other applications. Additionally, use AWS Identity and Access Management (IAM) policies to define which users and roles can access the application. You can also specify limits to control and track usage costs incurred by the application.

Consider creating multiple applications when you must do the following:

Use different open source frameworks

Use different versions of open source frameworks for different use cases

Perform A/B testing when upgrading from one version to another

Maintain separate logical environments for test and production scenarios

Provide separate logical environments for different teams with independent cost controls and usage tracking

Separate different line-of-business applications

EMR Serverless is a Regional service that simplifies how workloads run across multiple Availability Zones in a Region. To learn more about how to use applications with EMR Serverless, refer to Interact with and configure an EMR Serverless application .

Job run

A job run is a request submitted to an EMR Serverless application that the application asychronously executes and tracks through completion. Examples of jobs include a HiveQL query that you submit to an Apache Hive application, or a PySpark data processing script that you submit to an Apache Spark application. When you submit a job, you must specify a runtime role, authored in IAM, that the job uses to access AWS resources, such as Amazon S3 objects.

You can submit multiple job run requests to an application, and each job run can use a different runtime role to access AWS resources. An EMR Serverless application starts executing jobs as soon as it receives them and runs multiple job requests concurrently. To learn more about how EMR Serverless runs jobs, refer to Running jobs .

Workers

An EMR Serverless application internally uses workers to execute your workloads. The default sizes of these workers are based on your application type and Amazon EMR release version. When you schedule a job run, override these sizes.

When you submit a job, EMR Serverless computes the resources that the application needs for the job and schedules workers. EMR Serverless breaks down your workloads into tasks, downloads images, provisions and sets up workers, and decommissions them when the job finishes. EMR Serverless automatically scales workers up or down based on the workload and parallelism required at every stage of the job. This automatic scaling removes the need for you to estimate the number of workers that the application needs to run your workloads.

Pre-initialized capacity

EMR Serverless provides a pre-initialized capacity feature that keeps workers initialized and ready to respond in seconds. This capacity effectively creates a warm pool of workers for an application. To configure this feature for each application, set the initial-capacity parameter of an application. When you configure pre-initialized capacity, jobs can start immediately so that you can implement iterative applications and time-sensitive jobs. To learn more about pre-initialized workers, refer to Configuring an application when working with EMR Serverless .

EMR Studio

EMR Studio is the user console for managing your EMR Serverless applications. If an EMR Studio doesn't exist in your account when you create your first EMR Serverless application, we automatically create one for you. Access EMR Studio either from the Amazon EMR console, or turn on federated access from your identity provider (IdP) through IAM or IAM Identity Center.

When you do this, users can access Studio and manage EMR Serverless applications without direct access to the Amazon EMR console. To learn more about how EMR Serverless applications works with EMR Studio, refer to Creating an EMR Serverless application from the EMR Studio console and Running jobs from the EMR Studio console .

Document Conventions Prerequisites for getting started Did this page help you? - Yes Thanks for letting us know we're doing a good job! If you've got a moment, please tell us what we did right so we can do more of it. Did this page help you? - No Thanks for letting us know this page needs work. We're sorry we let you down. If you've got a moment, please tell us how we can make the documentation better.

Fonte 1: Documentação do Amazon EMR Serverless (https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html)

What is Amazon EMR Serverless? - Amazon EMR

View a markdown version of this page What is Amazon EMR Serverless? - Amazon EMR Documentation Amazon EMR Documentation Amazon EMR Serverless User Guide Concepts What is Amazon EMR Serverless? Amazon EMR Serverless is a deployment option for Amazon EMR that provides a serverless runtime environment. This simplifies the operation of analytics applications that use the latest open-source frameworks, such as Apache Spark and Apache Hive. With EMR Serverless, you don’t have to configure, optimize, secure, or operate clusters to run applications with these frameworks. EMR Serverless helps you avoid over- or under-provisioning resources for your data processing jobs.

EMR Serverless automatically determines the resources that the application needs, gets these resources to process your jobs, and releases the resources when the jobs finish. For use cases where applications need a response within seconds, such as interactive data analysis, you can pre-initialize the resources that the application needs when you create the application. With EMR Serverless, you continue to get the benefits of Amazon EMR, such as open source compatibility, concurrency, and optimized runtime performance for popular frameworks. EMR Serverless is suitable for customers who want ease in operating applications using open source frameworks. It offers quick job startup, automatic capacity management, and straightforward cost controls.

Concepts

In this section, we cover EMR Serverless terms and concepts that appear throughout our EMR Serverless User Guide.

Release version

An Amazon EMR release is a set of open-source applications from the big data ecosystem. Each release includes different big data applications, components, and features that you select for EMR Serverless to deploy and configure so that they can run your applications. When you create an application, specify its release version. Choose the Amazon EMR release version and the open source framework version that you want to use in your application. To learn more about pre-release versions, refer to Amazon EMR Serverless release versions .

Application

With EMR Serverless, you can create one or more EMR Serverless applications that use open source analytics frameworks. To create an application, specify the following attributes:

The Amazon EMR release version for the open source framework version that you want to use. To determine your release version, refer to Amazon EMR Serverless release versions .

The specific runtime that you want your application to use, such as Apache Spark or Apache Hive.

After you create an application, submit data-processing jobs or interactive requests to your application.

Each EMR Serverless application runs on a secure Amazon Virtual Private Cloud (VPC) strictly apart from other applications. Additionally, use AWS Identity and Access Management (IAM) policies to define which users and roles can access the application. You can also specify limits to control and track usage costs incurred by the application.

Consider creating multiple applications when you must do the following:

Use different open source frameworks

Use different versions of open source frameworks for different use cases

Perform A/B testing when upgrading from one version to another

Maintain separate logical environments for test and production scenarios

Provide separate logical environments for different teams with independent cost controls and usage tracking

Separate different line-of-business applications

EMR Serverless is a Regional service that simplifies how workloads run across multiple Availability Zones in a Region. To learn more about how to use applications with EMR Serverless, refer to Interact with and configure an EMR Serverless application .

Job run

A job run is a request submitted to an EMR Serverless application that the application asychronously executes and tracks through completion. Examples of jobs include a HiveQL query that you submit to an Apache Hive application, or a PySpark data processing script that you submit to an Apache Spark application. When you submit a job, you must specify a runtime role, authored in IAM, that the job uses to access AWS resources, such as Amazon S3 objects.

You can submit multiple job run requests to an application, and each job run can use a different runtime role to access AWS resources. An EMR Serverless application starts executing jobs as soon as it receives them and runs multiple job requests concurrently. To learn more about how EMR Serverless runs jobs, refer to Running jobs .

Workers

An EMR Serverless application internally uses workers to execute your workloads. The default sizes of these workers are based on your application type and Amazon EMR release version. When you schedule a job run, override these sizes.

When you submit a job, EMR Serverless computes the resources that the application needs for the job and schedules workers. EMR Serverless breaks down your workloads into tasks, downloads images, provisions and sets up workers, and decommissions them when the job finishes. EMR Serverless automatically scales workers up or down based on the workload and parallelism required at every stage of the job. This automatic scaling removes the need for you to estimate the number of workers that the application needs to run your workloads.

Pre-initialized capacity

EMR Serverless provides a pre-initialized capacity feature that keeps workers initialized and ready to respond in seconds. This capacity effectively creates a warm pool of workers for an application. To configure this feature for each application, set the initial-capacity parameter of an application. When you configure pre-initialized capacity, jobs can start immediately so that you can implement iterative applications and time-sensitive jobs. To learn more about pre-initialized workers, refer to Configuring an application when working with EMR Serverless .

EMR Studio

EMR Studio is the user console for managing your EMR Serverless applications. If an EMR Studio doesn't exist in your account when you create your first EMR Serverless application, we automatically create one for you. Access EMR Studio either from the Amazon EMR console, or turn on federated access from your identity provider (IdP) through IAM or IAM Identity Center.

When you do this, users can access Studio and manage EMR Serverless applications without direct access to the Amazon EMR console. To learn more about how EMR Serverless applications works with EMR Studio, refer to Creating an EMR Serverless application from the EMR Studio console and Running jobs from the EMR Studio console .

Document Conventions Prerequisites for getting started Did this page help you? - Yes Thanks for letting us know we're doing a good job! If you've got a moment, please tell us what we did right so we can do more of it. Did this page help you? - No Thanks for letting us know this page needs work. We're sorry we let you down. If you've got a moment, please tell us how we can make the documentation better.

Fonte: Documentação do Amazon EMR Serverless