Cloud Cost Optimisation Dashboard (FinOps)

The Problem and Why It Mattered

I built a real-time cloud cost observability and optimisation system — Billing Export to BigQuery as the data foundation, cost attribution by team, anomaly detection for spend spikes, and a Looker Studio dashboard for engineering and finance. The relevant technical and business risk is the cost of leaving this workflow, control, or product boundary unclear, manual, insecure, or difficult to operate. The case study focuses on the decisions used to make that risk manageable.

Objectives and Success Criteria

• Deliver the capability described in the project objective using a maintainable technical design.
• Make the main workflow, control boundary, and operational responsibilities explicit.
• Produce reviewable implementation evidence without publishing confidential values or customer data.

Requirements and Constraints

The implementation had to be reproducible, observable, and safe to describe publicly without exposing operational secrets or private infrastructure detail.

My Role and Ownership

I owned the technical design and implementation described in this portfolio record, including the technology choices, core system boundaries, and the documentation of the resulting approach. Related organisational decisions, vendor operations, and formal certification decisions remain outside the scope of this technical case study unless stated explicitly.

Solution Overview

The solution is built around BigQuery, Looker Studio, Cloud Functions. Its implementation is organised as the components below so that the architecture, control points, and operating model can be evaluated separately rather than treated as a list of tools.

Architecture and System Design

The public architecture description follows the path from the initiating user, workload, or operator action through the relevant cloud engineering components. Access boundaries, data or state handling, and operational controls are described at a safe level; secrets, private addresses, and client-sensitive implementation details are intentionally omitted.

Technology Decisions and Trade-Offs

I selected BigQuery, Looker Studio, Cloud Functions because they support the project objective and make the critical engineering boundaries explicit. The trade-off is operational and platform complexity: the design depends on documented configuration, controlled access, and ongoing review rather than assuming that a managed service or framework removes engineering responsibility.

Implementation Walkthrough

The following implementation notes describe the components that matter most to the project outcome. They are intentionally focused on technical decisions and operating behaviour instead of reproducing entire source files or sensitive configuration.

Implementation: Billing Data Pipeline

GCP Billing Export streams detailed usage and cost data into BigQuery in near-real-time. A transformation layer produces daily partitioned summary tables by project, service, SKU, and label.

Implementation: Cost Attribution Model

Label taxonomy (team, environment, product, cost-centre) enforced via Organisation Policy. The attribution model answers: 'How much did the payments service cost in staging last month?' — a question most teams can't answer without this infrastructure.

Implementation: Anomaly Detection

A Python Cloud Function runs daily, compares the previous day's costs to the 30-day rolling average, and alerts in Slack if spend is more than 2 standard deviations above the mean. Caught a runaway BigQuery query that scanned 4TB in a single day within 24 hours.

Implementation: Committed Use Discount Analysis

A BigQuery model identifies workloads that have run consistently for 30+ days and calculates the projected saving from purchasing CUDs. Output is a prioritised list ranked by ROI.

Implementation: Results

Identified and eliminated $18,000/year in idle resource spend in the first month. Achieved 34% cost reduction on Compute Engine through CUD purchases. Cost anomaly detection caught 4 incidents in 6 months.

Challenges, Failures and Corrections

This portfolio record does not invent deployment failures, client incidents, or corrective actions that are not supported by the published project evidence. Where operational details are confidential or were not captured for public release, the case study documents the design boundary and lesson instead of presenting an unsupported narrative.

Security, Compliance and Risk

The public record avoids secrets, customer data, and private infrastructure details. The relevant security controls, access boundaries, and operational risks are addressed in the implementation notes where they are material to the project.

Testing, Validation and Evidence

The evidence published for this project is the documented implementation, its stated architecture and controls, and the portfolio record. Identified and eliminated $18,000/year in idle resource spend in the first month. Achieved 34% cost reduction on Compute Engine through CUD purchases. Cost anomaly detection caught 4 incidents in 6 months. Claims are limited to the implementation details and outcomes documented in this case study.

Results and Evidence

Identified and eliminated $18,000/year in idle resource spend in the first month. Achieved 34% cost reduction on Compute Engine through CUD purchases. Cost anomaly detection caught 4 incidents in 6 months.

What I Learned

This project reinforced that cloud engineering work is not a technology checklist. The durable value comes from making boundaries, trade-offs, ownership, and evidence clear enough for another engineer or stakeholder to review and operate the system responsibly.

What I Would Improve Next

• Expand automated validation around the most important workflow and control boundaries.
• Add or deepen operational evidence, such as dashboards, test artefacts, or an architecture diagram where it is safe to publish one.
• Review the design against the next expected scale, reliability, and governance requirement before extending the system.

Final Takeaway

This cloud engineering case study addresses a defined technical problem through BigQuery, Looker Studio, Cloud Functions. It demonstrates my ability to turn requirements into an explainable design, implement the important controls, and document outcomes and limitations without overstating what the evidence proves.