Early-stage dataset marketplace startup
Threat modeling and scan gates for an enterprise data marketplace, set before features
When a dataset marketplace moved toward enterprise buyers, we made security evidence the first deliverable: threat modeling, infrastructure as code and scanning gates came before features. Organization roles, business verification, encryption between machines and attestation were built on that base.
Data & AnalyticsEnterprise dataset marketplaceModernized a live product
The challenge
Moving the marketplace toward enterprise buyers meant organization roles, business verification tied to plans and a security architecture the client saw as critical to its value.
What we did
We opened the rebuild with threat modeling, a scanning pipeline and infrastructure as code, then built the marketplace core and a Rust and Tauri agent that encrypts and attests datasets locally.
The outcome
The first two milestones, architecture and foundations and the marketplace core, were delivered and reported to the client's CTO, with encryption between machines verified on two physical devices.
- First milestone
- Architecture, threat model, security pipeline
- Second milestone
- Organization roles and the marketplace core
- Pipeline gates
- SAST and DAST with Snyk, SonarQube, OWASP ZAP
- Local agent
- Rebuilt in Rust and Tauri, verified on two machines
The starting point
A dataset marketplace where files stay with the seller and only manifests reach the platform, built with us in a first version. Its next version targeted enterprise buyers and organizations, with a local validation agent and a third-party trust and encrypted transfer layer.
What had to change
Enterprise use needed organization roles, buyer and seller verification tied to subscription plans, and watermarking and attestation for shared data. The client saw security architecture as critical to its value, so it had to be designed in, not added later.
What could not break
- Validation, watermarking and encryption stay on the user's machine
- Encryption must work between a seller's and a buyer's machines
- Buyers and sellers pass KYC or KYB, tied to their subscription tier
- Every change goes through static and dynamic security scans
The calls we made
The client saw its security architecture as central to what it sells, so we treated security evidence as the first deliverable and built the marketplace on top of it.
- 01
Security before features
Threat modeling and the design of the scanning pipeline sat in the first milestone, and SOC 2 evidence collection started with the marketplace core, so every later change runs through them.
- 02
Keep the work on the user's machine
The agent parses, scores, watermarks, encrypts and attests datasets on the user's machine, which keeps the non-custodial design the marketplace was founded on as it moves to organizations.
- 03
Rust and Tauri for the local agent
The agent now carries encryption, watermarking and attestation, with secure IPC between its parts, so we built it in Rust on Tauri v2.
- 04
Infrastructure as code
Terraform modules for networking, KMS, S3, RDS, ElastiCache, Secrets Manager, IAM, ALB and ECS make every environment reviewable and repeatable.
How the system fits together
What we built
Enterprise marketplace core
Next.js and FastAPI with JWT and cookie auth with token rotation, TOTP 2FA, Google and LinkedIn login, five organization roles from Owner to Viewer, and Stripe KYC and KYB tied to a five tier subscription model.
Security foundation
Threat modeling, a GitHub Actions pipeline with SAST and DAST gates from Snyk, SonarQube and OWASP ZAP, branch protection, CloudWatch monitoring and SOC 2 evidence collection.
Local validation agent
A Rust and Tauri v2 agent with secure IPC, device provisioning and HMAC auth against the trust layer, AES-256-GCM encryption between machines and SHA3-256 attestation.
Watermarks and quality scores
Three watermarking approaches, limited data, omitted rows and masking, with a live preview, plus CSV, TSV, JSON and JSONL parsers with type inference and quality scoring.
Infrastructure as code
Terraform modules for networking, KMS, S3, RDS, ElastiCache, Secrets Manager, IAM, ALB and ECS, and a Playwright test framework.
How it works
Join as an organization
Buyers and sellers pass KYC or KYB through Stripe, pick a subscription tier and work under organization roles.
Provision the device
The local agent is provisioned to the user's machine and authenticates to the trust layer with HMAC.
Parse and score
The agent parses the dataset, infers its types and scores its quality on the user's machine.
Watermark and attest
The seller picks a watermarking approach with a live preview, and the agent creates a SHA3-256 attestation.
Encrypt and list
The dataset is encrypted with AES-256-GCM for the transfer between machines, and the listing goes up through encrypted endpoints with signed URLs.
What changed
The first two milestones, architecture and foundations and the marketplace core, were delivered and reported to the client's CTO. The enterprise version starts with its threat model, scanning gates and SOC 2 evidence collection already in place.
- Architecture and foundations, marketplace core
- Delivered
- Verified across two physical machines
- Encryption
Under the hood
- Datics role
- Product built by Datics
- Security
- Threat modeling, SAST and DAST gates with Snyk, SonarQube and OWASP ZAP, branch protection and SOC 2 evidence collection.
- Deployment
- Terraform modules for AWS networking, ECS Fargate, RDS, ElastiCache, KMS, S3, Secrets Manager, IAM and ALB, with CloudWatch monitoring.
- Integrations
- Stripe for KYC and KYB, a third-party trust and encrypted transfer layer with HMAC auth, and Google and LinkedIn login.
- Agent
- Rust on Tauri v2 with secure IPC, AES-256-GCM encryption, SHA3-256 attestation and parsers for CSV, TSV, JSON and JSONL.
Data and retrieval
Cloud and infrastructure
Apps and APIs
Also in the stack
How it shipped
Scoping the enterprise version
After the first version, a scoping round defined the enterprise marketplace, the local validation agent and the integration with the trust and transfer layer.
Milestone 1, architecture and foundations
ERD and service boundaries, auth with token rotation and TOTP 2FA, organization roles, the Rust and Tauri agent baseline, a Terraform VPC, GitHub Actions, threat modeling and the security pipeline design.
Milestone 2, marketplace core
KYC and KYB tied to subscription tiers, encrypted listing endpoints, device provisioning, cross-machine encryption, watermarking, attestation, parsers, Terraform modules, Playwright and the start of SOC 2 evidence collection.
Milestone reports
Each milestone closed with a written report to the client's CTO on what was built and verified, with a sprint summary in between.
Verified on real machines
Cross-machine encryption and decryption were checked on two physical machines before they were reported as done.
What this unlocked
The plan's later milestones add AI validation on this base: checks across several models, quality scoring, personal data checks and escalation to manual review.
Related work
Have a product with the same kind of constraints?
Tell us what already exists and what you want it to do. We will tell you what it takes.