Big Cloud

06 Oct, 2025

I recently came across a take-home exercise for a Product Security role at a large software company. During the exercise, I quickly discovered interesting challenges that forced me to think "from source".

Before diving into the details, I'll cover the core requirements of the exercise. The exercise focused on securing a new public-facing web application hosted on AWS. The key goals were to:

• Deploy the application in a scalable AWS environment & expose it publicly through an AWS Application Load Balancer or AWS CloudFront (CDN).

• Implement a reusable AWS WAF (Web Application Firewall) module with custom and managed rules.

• Develop a rapid mitigation script for quickly blocking suspicious traffic.

• Integrate a CI/CD pipeline that performs SAST on IaC components.

Infra

During infrastructure deployment, I utilized Terraform to define and manage all AWS resources (except the AWS WAF, more on this later) as code. I opted to deploy the application on an ECS cluster utilizing EC2 instances, giving me full scaling configurations while keeping the deployment straightforward. I exposed the application publicly through an AWS Application Load Balancer (ALB), which provided a reliable entry point to route incoming traffic to the EC2 instances.

AWS WAF, CDK & Race Conditions

WAF WebACL's?

A Web Application Firewall (WAF) Web Access Control List (WebACL) in AWS is a set of rules that protects web applications by filtering and monitoring HTTP(S) requests. It helps block common attack patterns like SQL injection, cross-site scripting (XSS), and other malicious traffic before they can reach application endpoints. A WebACL can be associated with resources such as CloudFront distributions or Application Load Balancers (ALB) to enforce these security rules at the edge. More info.

AWS Go CDK

You might be wondering: why not just deploy all resources utilizing Terraform? While Terraform is powerful for infrastructure, I’ve recently been diving deep into the Go ecosystem. I wanted to move beyond configuration files and actually build within the CLI frameworks that Go excels at.

A crucial part of the implementation was associating the WebACL with the front-door resource; either a CloudFront distribution or an Application Load Balancer (ALB). This required passing the resource ARN during deployment to properly link the WebACL.

contextValue := app.Node().TryGetContext(jsii.String("resource_arn"))

	// Check if the value exists and is of the expected type (string).
	resourceArnStr, ok := contextValue.(string)
	
	if !ok || resourceArnStr == "" {
		log.Fatalf("Context value 'resource_arn' not provided or not a string. Please use `cdk deploy -c resource_arn=\"arn:aws:elasticloadbalancing:...\"`")
	}

One of the trickiest challenges I faced was a race condition during deployment. Since the WebACL resource had to be created and then associated with an existing front door resource, the deployment sometimes failed or behaved inconsistently if the association happened before the resource was fully available.

Eventual Consistency

In AWS CDK, the app.Synth() function is responsible for generating the CloudFormation templates from your application’s constructs. This step translates your code into actual deployable infrastructure definitions.

In most CDK supported languages (e.g., TypeScript or Python), app.synth() is implicitly called at the end of execution. So explicitly calling it again typically has no effect.

However, in Go, where you're working more directly with lower level constructs, that implicit behavior isn't always guaranteed; especially in more complex setups or when customizing the synthesis phase.

To resolve the issue, I explicitly forced the synthesis step using:

app.Synth(&awscdk.StageSynthesisOptions{
	Force: jsii.Bool(true),
})

As a result, I was able to include and deploy AWS-managed rule groups and few custom rules as part of the WAF module without race conditions or partial deployments. More info.

WAF Testing

The deployed application is vulnerable by default and widely used for educational purposes. This allows me to test known vulnerable endpoints with malicious payloads. Below is an example an SQL Injection attempt.

As the terminal output displays, the request was responded 200. Passing the parameter

qwert')) UNION SELECT id, email, password, '4', '5', '6', '7', '8', '9' FROM Users--

to a specific endpoint allows us to verify the vulnerabilities that exist before associating the WAF with the Application Load Balancer. Once the association has been completed, request respond 403.

Rapid Updates

The last major component of the exercise involved creating a script for rapid updates to AWS WAF configurations in response to emerging threats. The purpose is to allow infra/security teams to quickly block suspicious IPs, user-agent patterns, or malicious payloads without waiting for full infrastructure re-deployments.

I chose to build an CLI application using the Cobra framework in Go, which made it easy to create a structured command-line interface with support for subcommands and flags. The tool supports multiple types of rules, such as:

--ip: Block or allow specific IPs or CIDR blocks

--regex: Match suspicious patterns in headers, query strings, or bodies

--file: Load bulk entries (e.g., IPs or regex rules) from a file

check it out here!.

While testing the functionality of the application, I constantly ran into database update latency issues. I realized on-premises or in the "Cloud", systems are systems.

Distributed Systems & The Illusion of Instant Consistency

In modern cloud environments, we often assume things should “just work”, infrastructure should be created instantly, API calls should be atomic, and deployments should complete without surprises. But when you're dealing with distributed systems, these assumptions can be dangerous.

This exercise exposed that tension clearly. I plan to dive deeper into application security, distributed systems, and infra/security automation workflows.