Semantics in Flutter - under the hood

Findings:

1. SemanticsNode already has a Key in it!
   1. Could it just be uploaded to the engine?
   2. Nah, because this key is associated with the Semantics itself???
2. PRs that added support for tooltip in semantics. Something similar could be done with identifier.
   1. framework part https://github.com/flutter/flutter/pull/87684
   2. engine part
      1. original https://github.com/flutter/engine/pull/27893
   3. relanded https://github.com/flutter/engine/pull/28211
3. PR that optimized semantics. Has some stats about how many “semantics” there are in big apps.

How does semantics flow from the Framework to the Engine?

In the framework

1. Developer creates a Semantics widget, passes SemanticsProperties to it
2. The Semantics widget creates its render object – RenderSemanticsAnnotations
3. The RenderSemanticsAnnotations render object calls markNeedsSemanticsUpdate(). This method is inherited from the RenderObject abstract class
4. markNeedsSemanticsUpdate() adds the render object to PipelineOwner._nodesNeedingSemantics.

Also, every RenderObject manages its own SemanticConfiguration.

5. when somebody wants the RenderObject to tell them about its semantics, it calls describeSemanticsConfiguration and passes (TODO)

At some later point, a new frame is scheduled by the engine. This is handled by RendererBinding in the persistent frame callback that it registered. This callback calls RendererBinding’s drawFrame(), which drives the frame rendering pipeline. In one of the last stages of this pipeline, it callsPipelineOwner.flushSemantics(). This gathers _nodesNeedingSemantics.

Finally, SemanticsOwner.sendSemanticsUpdate() is called. This function does (quite some stuff), but most importantly, it finally calls SemanticsUpdateBuilder.build() to obtain a SemanticsUpdate.

💡 SemanticsUpdateBuilder is an abstract class from the engine. It’s implemented in C++ (header) (source).

Then, the SemanticsOwner.onSemanticsUpdate callback is called and the SemanticsUpdate is passed to it as an argument.

(omitted some calls)

Then that SemanticsUpdate is passed to the engine by calling RenderView.updateSemantics().

In the engine

TODO, but generally:

• first, the SemanticsUpdate lands in Engine’s C++ code, and then
• it is delivered to the appropriate embedder APIs
• on Android, JNI is heavily used

Naming observations

• Methods that start with “handle”, e.g. handlePlatformBrightnessChanged, are callbacks to be run when the engine says so

2024-05-11 – GitHub Actions beg for a supply chain attack
2024-03-19 – Writing a custom Dart VM service extension (part 1)
2024-02-08 – On using smartphone for things that make sense
2023-11-30 – Semantics in Flutter - under the hood
2023-11-25 – Flutter Engine notes
2023-09-17 – Creating and managing Android Virtual Devices using the terminal
2023-05-27 – Suckless Android SDK setup
2023-05-26 – Let’s start over
2023-05-21 – Short thought on “The Zen of Unix”
2023-05-15 – Notes about “flutter assemble”