Senior Data Platform Engineer

OVERVIEW

Reporting to the VP, Technical Operations, the Senior Data Platform Engineer is a senior, hands-on individual contributor on our Engineering, Data Platform team. This is a full-time, remote role open to candidates based in the United States or Canada, with no direct reports.

StoreForce helps specialty retailers run their stores, from workforce scheduling to performance analytics, across tens of thousands of locations worldwide. We are building a new data platform on Azure Databricks that turns the operational data behind those stores into governed metrics, embedded analytics, and AI-driven insight for our customers.

As a Senior Data Platform Engineer, you will build, run, and scale that platform: Spark and Delta Lake pipelines from our multi-tenant SQL Server estate, Unity Catalog governance with a shared semantic layer, and the analytics agent on top. You will work in a small platform team, alongside an implementation partner during the initial build, and own the platform’s growth from the first customers to the full base.

WHAT YOU'LL DO

1. Lakehouse Engineering & Data Pipelines

• Build and run the lakehouse on Azure Databricks, including change tracking and change data capture (CDC) ingestion from SQL Server through Lakeflow Connect.
• Develop Spark and Delta Lake transformations in Lakeflow Spark Declarative Pipelines and jobs, along with orchestration and serving layers.
• Implement and maintain the medallion and dimensional data model in Delta Lake.
• Tune Spark jobs, Photon, liquid clustering, and storage for performance and cost.
• Measure page and report latency against Lakebase and Databricks SQL, adding a columnar serving store only if the numbers require it.

2. Semantic Layer & Data Governance

• Maintain the governed semantic layer in Unity Catalog (Metric Views) so that dashboards, embedded analytics, applications, and AI agents consume the same certified metrics.
• Enforce per-tenant row and column security at query time.
• Implement StoreForce’s retail KPI logic in Metric Views, including fiscal calendars, 53-week comparisons, comp days, and targets.
• Reconcile Metric View results to the existing application’s numbers.

3. Platform Scale & Tenant Onboarding

• Scale the platform by configuration: onboard tenants and add KPIs by repeating the established pattern.
• Map each tenant’s configurable KPI columns to one governed schema.
• Extend and fix the pattern when it needs a new code path.

4. Data Quality, Reliability & Cost

• Enforce data quality in the pipeline: expectations, data contracts, reconciliation to source, and freshness and volume checks.
• Maintain lineage and Lakehouse Monitoring, treating data incidents like production incidents.
• Keep the platform reliable in production, favoring serverless and managed services over undifferentiated infrastructure.
• Deliver everything as code through CI/CD, with close attention to compute sizing and DBU spend.

5. AI-Powered Analytics

• Build and operate the LLM-powered analytics agent that answers natural-language questions against the semantic layer.
• Use Databricks AI tooling alongside Anthropic Claude and OpenAI models.
• Ensure tenant isolation, runtime guardrails, and an automated evaluation suite that acts as a quality gate.
• Use AI-assisted development to ship quickly and effectively.

6. Collaboration & Platform Ownership

• Work alongside the implementation partner during the initial build and take ownership of the platform at handover.
• Serve as the go-to engineer for data at StoreForce, contributing design decisions backed by evidence.
• Keep engineering standards and runbooks current.
• Support the teams consuming the platform and triage data incidents.

Other duties as assigned by your manager and senior leadership.

QUALIFICATIONS
Competencies Required:

• Technical Depth - Hands-on expert across the Databricks and Spark core who writes production code every day
• Ownership - Takes end-to-end accountability for the platform, from build through production support
• Data Quality Mindset - Treats data correctness and data incidents with the rigor of production software
• Evidence-Based Decisions - Measures before deciding and backs design choices with data
• Learning Agility - Picks up new tools quickly and embraces AI-assisted development
• Collaboration & Communication - Works effectively with partners and consuming teams; explains trade-offs clearly to technical and non-technical audiences

EXPERIENCE / KNOWLEDGE

• 5+ years building data pipelines or backend systems, including at least 2 years running a modern data stack in production
• Hands-on production experience with Databricks: Unity Catalog, Delta Lake, Lakeflow Jobs and Spark Declarative Pipelines (or their predecessors, Workflows and Delta Live Tables), Databricks SQL, and deploying through Databricks Asset Bundles or an equivalent as-code practice
• Strong Apache Spark skills: PySpark and Spark SQL, Structured Streaming, reading query plans, diagnosing shuffle and skew, and tuning jobs and compute for performance and cost
• Strong SQL (window functions, complex joins, query optimization) and strong Python for data work
• Solid data modeling across medallion, dimensional, and incremental patterns, with experience tuning Delta tables (liquid clustering, change data feed, OPTIMIZE and VACUUM strategy)
• Experience implementing data quality and observability (expectations, contracts, reconciliation, lineage, monitoring, alerting)
• Hands-on experience with modern LLM providers (Anthropic Claude and OpenAI), building AI features against their APIs
• Fluent in modern software engineering practices (version control, automated testing, CI/CD) and able to explain everyday trade-offs such as serverless vs. classic compute or streaming vs. batch; infrastructure or DevOps specialization is not required.

PREFERRED QUALIFICATIONS

• Databricks Certified Data Engineer Associate or Professional
• Experience deploying LLMs against structured data in production on Databricks (Genie, the agent framework, Model Serving for external models, MLflow evaluation), particularly evaluation, guardrails, and CI gating
• Experience with multi-tenant SaaS data on Databricks, including per-tenant isolation with Unity Catalog row filters and column masks, and catalog design for many tenants
• Experience ingesting from SQL Server and Azure SQL Managed Instance with Change Tracking or CDC over private networking
• Experience serving curated data to always-on applications from Lakebase or PostgreSQL
• Experience delivering embedded analytics inside a product, not only internal business intelligence
• Experience taking over a platform from a consultancy or another team and running it well afterward.

TECHNICAL ENVIRONMENT

• Ingestion - Lakeflow Connect with SQL Server change tracking
• Storage - Delta Lake on Azure Data Lake Storage Gen2 in StoreForce’s own tenancy, in open formats so the data outlives any engine
• Transformation - PySpark, Spark SQL, and Lakeflow Spark Declarative Pipelines
• Governance - Unity Catalog for governance, lineage, and the semantic layer (Metric Views)
• Orchestration & Compute - Lakeflow Jobs, with serverless compute where it fits
• Serving - Databricks SQL for BI and embedded analytics; Lakebase or PostgreSQL as the always-on serving store
• Infrastructure & Security - Everything as code with Databricks Asset Bundles and Terraform; private networking with Entra ID throughout

You don’t need experience with every technology: bring hands-on depth across the Databricks and Spark core and be ready to learn the rest. This is a data engineering role, not infrastructure.

KEY PERFORMANCE INDICATORS (KPIs)

• Pipeline success rates, data freshness against agreed SLAs, and availability of serving layers
• Reconciliation of governed KPIs to the existing application’s numbers, data quality check pass rates, and number and severity of data incidents
• Time to detect and resolve data incidents, with root causes addressed and runbooks updated
• Dashboard, report, and embedded analytics latency against agreed targets
• Time to onboard a new tenant and to add a new KPI using the standard pattern
• Compute and DBU spend per tenant, and cost trends against budget
• Evaluation suite pass rates, answer accuracy, guardrail effectiveness, and zero cross-tenant data exposure
• Successful handover from the implementation partner, current standards and runbooks, and satisfaction of teams consuming the platform.

WHY WORK WITH US AT STOREFORCE

StoreForce is growing quickly – it’s the kind of place you can make a real impact. We thrive on change, collaboration, and trust; And are fueled by a mutual passion for Specialty Retail. At StoreForce you’ll have the chance not only to work with some amazing people, but also with some of the world’s best retail brands.

• We're remote and offer flexibility in your schedule
• Employer-paid health benefits including dental, prescription drug coverage, and a Health Spending Account;
• RRSP Matching with First Time Home Buyer support*
• Company paid training and professional development
• Extended Long Weekends and Religious Observance Days
• Employee Referral Program
• Spotlight Recognition Bonus Program
• Dedicated Employee Assistance Program (EAP)
• Time off to actively support a charity/volunteer
• President's Club

*Speak with HR for full program details.

StoreForce is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind. We are committed to the principle of equal employment opportunity for all individuals and to providing individuals with a work environment that is diverse and inclusive. This starts by welcoming and encouraging applications from all individuals including those with disabilities. Accommodations are available on request for candidates taking part in all aspects of the selection process.

At StoreForce, we leverage AI tools to support a fair and efficient recruitment process. This may include AI-assisted resume screening to help identify top candidates and AI note taking during interviews. All hiring decisions are made by real people. AI is used only as a supplemental tool to enhance, not replace, human judgment. We never accept or respond to speculative applications from third party recruiters and work only with preferred partners.