Google moves federated training into protected servers with inspectable privacy rules

Private uploads still left a server trust gap

Federated learning (FL) lets clients collaborate on model training under a service provider’s coordination. In Google Research’s definition, clients should control their data, which workloads may use it, and how those workloads anonymize it. The goal is broader than deciding where training computation happens.

Earlier systems left a gap between intended privacy behavior and behavior outsiders could check. Devices uploaded data for immediate combination, but external observers could not establish that the server had never inspected or logged those uploads. A promise about handling data was not itself evidence that the server followed it.

Secure Aggregation, a cryptographic method for protecting uploads, addressed part of that problem. The authors say it did not work with the strongest central differential privacy guarantees they discuss. Differential privacy (DP) is the anonymization guarantee used for trained models. Earlier servers added random noise to sums of training updates, but devices and auditors could not verify that step.

Training also faced a separate practical limit: it depended on available devices and their computing resources. Multiple workloads competed for those resources. Katharine Daly and Daniel Ramage of Google Research describe a replacement intended to improve both privacy verification and training execution. Their account reports a working deployment, rather than only a proposed design.

Make server processing inspectable rather than merely promised

The central idea is to move training computation into Trusted Execution Environments (TEEs) on servers. These are protected environments designed to conceal their internal state and prevent interference with their logic. They also let outside parties verify remotely what logic executes. The authors explicitly qualify these properties by the limits of current TEE hardware.

Moving data to a server therefore does not mean giving its operator ordinary access to that data. Uploaded training data stays encrypted until an approved program processes it inside a TEE. Workload operators see only metrics and model weights produced with differential privacy, rather than the underlying training data.

Access policies, descriptions of the server programs permitted to use an upload, connect device participation to that protected computation. Participating devices know the complete set of workloads that may access their uploads. Auditors can inspect the published policies to track that set too. The policy is therefore part of the privacy mechanism, not just explanatory documentation.

This changes what must be trusted. Earlier systems required trust that the server operator executed privacy steps correctly. The new design makes the permitted training logic inspectable and restricts data processing to protected execution. It is a step toward removing operator trust, not a claim that hardware limitations or software correctness concerns have disappeared.

Follow an upload through the training system

The concrete application in the post is Gboard next-word prediction in English and Japanese. The authors do not trace an individual training record. Instead, the system description explains how device uploads become inputs to a protected training workload and which parts of that process outsiders can inspect.

1. Publish the permitted programs. Access policies describe the Python training program and appear in Rekor, a public log that external auditors can inspect. Devices know which server workloads could use their data before that data is processed.
2. Restrict access to encrypted uploads. Training data can be decrypted and used only inside TEEs executing programs covered by those policies. Permission also expires after a limited period following upload. The post does not specify that period’s duration.
3. Separate privacy rules from proprietary information. The TEE can load serialized information into the Python program during execution, a feature called sideloading. This allows model architecture and data preprocessing information to remain proprietary. All privacy-relevant logic must stay fixed in the inspectable Python program.
4. Collect uploads before executing training. Rather than making progress depend on daily changes in device availability, the system gathers the uploads first. During server execution, the program calculates a device participation schedule and uses it to adjust other DP parameters.
5. Compute training updates on protected servers. Client gradients are the training computations associated with client data that earlier systems performed on devices. Moving them to servers enables computation across many machines. Operators receive metrics and differentially private model weights.

The scheduling change and the privacy change support different parts of the design. Gathering uploads ahead of execution separates training progress from devices coming and going. Inspectable programs and protected execution govern what happens to those uploads afterward. Faster computation alone would not establish that the privacy rules were followed.

Deployment evidence is stronger than the measurement detail

The authors report that Gboard has launched English and Japanese next-word prediction models using the TEE-based system. They report improved accuracy and stronger privacy guarantees, alongside substantially faster computation than the previous FL system. These are production claims about an adopted system, not results from a benchmark table.

The concrete timing number describes the earlier approach: a model could require 1-2 months of training. Device availability, device computing limits, and competition between workloads constrained progress. The new system moves those constraints to the server and spreads computation across machines. Its current limiting resource, according to the authors, is TEE capacity.

The post gives no corresponding training duration for the new system. It also supplies no accuracy values or numerical DP settings. Readers therefore cannot calculate a speedup from the supplied evidence or judge the size of the accuracy improvement. The old duration is useful context, but it is not a measured before-and-after comparison.

The authors also describe improved device coverage as a goal of shifting computation to servers. However, the post provides no coverage measurements. The supported takeaway is that the system has reached deployment and removes the stated device-computation constraint, while the scale of its reported gains remains unspecified.

Inspectable execution is not a full correctness proof

The main qualification is that privacy relies on the protections current TEEs actually provide. The authors identify side-channel observations as an area needing further mitigation. They expect better hardware and continuing research to better protect workloads loaded during execution from malicious attacks on the server side. That stronger protection is a future expectation, not a completed result.

Sideloading has its own important condition. Keeping proprietary information out of the published program preserves auditability only if all privacy-relevant logic remains fixed in that program. The design does not give unrestricted runtime-loaded code a blanket privacy guarantee. Its stated guarantee depends on maintaining the separation between proprietary information and inspectable privacy behavior.

Verifying which program runs also differs from proving that its implementation is correct. The authors describe full correctness proofs for DP software and system components as something systems like theirs might eventually provide. The current account should therefore not be read as a completed mathematical proof covering every implementation component.

Finally, the evidence remains a high-level engineering account. It reports deployment and qualitative improvements, but not controlled comparisons or detailed measurements. Server execution also relocates resource limits rather than eliminating them: training still needs enough TEE resources. Both the protection boundary and the compute boundary matter when assessing whether this design fits another workload.

A useful design for engineers and privacy auditors

Engineers training on private device data should pay attention to the combination, rather than server-side computation alone. Published workload rules, protected execution, limited data access, and private model outputs work together. The design shows how moving computation off devices can coexist with a client-centered definition of federated learning, provided that control and verification remain explicit.

For privacy researchers and auditors, the useful distinction is between checking what runs and proving that everything it does is correct. This system strengthens the former while leaving full implementation proofs as future work. Its conditional treatment of proprietary runtime information is also central to evaluating the claimed privacy boundary.

The authors see larger models as a possible next use because moving client-gradient computation to servers lifts constraints imposed by on-device computing resources. They say integrating TEEs with accelerators will matter for that direction. For now, the clearest result is the Gboard deployment: inspectable server-side privacy rules are in use, while detailed performance evidence and stronger correctness guarantees remain to be supplied.