Engineering Blog

Improving Airflow UI Security

Airflow is a platform to programmatically author, schedule and monitor workflows (called directed acyclic graphs–DAGs–in Airflow). When we first adopted Airflow in late 2015, there were very limited security features. This meant that any user that gained access to the...

Using Linkerd as a Service Mesh Proxy at WePay

In the upcoming months, we are going to write a series of posts documenting WePay Engineering’s journey from traditional load balancers to a service mesh on top of Google’s Kubernetes Engine (GKE). In this first part of the series, we...

Configuration Management Framework at WePay

Configuration management is a common problem that needs to be solved by every company dealing with large scale deployments across many environments. Provisioning environments, deploying applications, and managing infrastructure service(s) is dependant on ensuring the correct configuration is available at...

WeTools: An Elixir Command Line Tool

Within WePay’s tools team, we are always looking for ways to make our engineers more productive. One of the challenging things we are trying to do is distribute a series of automated tasks to our developers, such as a way...

Data Backfilling at WePay

A perennial issue in the payments world is fraud. At WePay, we have our own fraud-detection system to deal with a variety of fraud issues. For example, when a merchant receives a payment of $50 from a credit card, our...

Splitting traffic with SplitIO

At WePay we are constantly evolving our core infrastructure to better meet the needs of our customers, whether that’s broadening the feature set for our end users, or delivering faster transactions for our payment processors. As part of scaling, WePay...

Simple Developer Docs using Jekyll, GitHub Pages, and GitHub Enterprise

I am a fan of simple solutions. In this post, I explain how we used Jekyll and GitHub Pages to solve our need for a new developer docs system. Like most startups, our first developer docs were just another controller...

Streaming databases in realtime with MySQL, Debezium, and Kafka

Change data capture has been around for a while, but some recent developments in technology have given it new life. Notably, using Kafka as a backbone to stream your database data in realtime has become increasingly common. If you’re wondering...

Sensu at WePay

My first project here at WePay was to replace our legacy monitoring system Check_MK, with something that we can easily configure and scale to meet the needs of our infrastructure and application services. Our monitoring needs have outgrown traditional tools...

Supporting Chip Cards at WePay

You have probably noticed that your credit cards now have little chips in them. You may have already used them at stores by inserting (or “dipping”) them into the card reader instead of swiping. While waiting for the transaction to...

Kafka Avro client

WePay uses Apache Kafka as its real time message broker service to publish and consume realtime events. Messages published to Kafka topics can adhere to a specific schema. To manage the schemas in a centralized location, we use Confluent’s schema-registry,...

Interviewing at WePay - The Why

In my last post, I consumed your valuable screen and brain space discussing what our interviewing process looks like - what we do, and what we are trying to achieve. This post (taking up more of your valuable screen real...

Training machine learning models with Airflow and BigQuery

WePay uses various machine-learning models to detect fraudulent payments and manage risk for payers, merchants and their platforms. The Problem In a previous blog post, our Data Science team described how we use a Random Forest algorithm to achieve an...

Loading data from Kafka into BigQuery with Kafka Connect

WePay recently released an open-source Kafka-BigQuery Connector on GitHub. We’ve decided to celebrate with a blog post detailing what exactly a Kafka Connector is, how we implemented ours, and why we needed one in the first place. Enjoy! Tracking events...

Building WePay's Webhook delivery system with Google Cloud Pub/Sub

Introduction Webhooks are user-defined HTTP callbacks. At WePay, we make use of webhooks we call Instant Payment Notifications or IPNs to update our partners on the status of transactions happening in our system. IPNs allow our partners to receive notifications...

Software Engineering Interviewing at WePay

What is this post about interviewing doing taking up valuable screen real estate on an engineering blog? Shouldn’t it be filled with discussions of cool new technology, code snippets, or coding how-to guides? Interviewing, like many things, is a problem...

BigQuery at WePay

Our previous posts provided an overview of our data warehouse, and discussed how we use Airflow to schedule our ETL pipeline. In this post, we’ll focus on how we’ve set up BigQuery as the database that powers our data warehouse....

Airflow at WePay

NOTE: We recently gave an Airflow at WePay talk to the Bay Area Airflow meetup group. The video and slides are both available. Our last post provided an overview of WePay’s data warehouse. In this post, we’ll be diving into...

Building WePay's data warehouse using BigQuery and Airflow

Over the coming weeks, we’ll be writing a series of posts describing how we’ve built and run WePay’s data warehouse. This series will cover our usage of Google Cloud Platform, BigQuery, and Apache Airflow (incubating), as well as how we...

How WePay mocks SFTP payment processor backends

In payments, as the volume of transactions increases, the number of ways things can go wrong also increases. This is particularly true at the point where a processor interacts with its bank acquirer. Here, declines can happen for a whole...