Semantics in Flutter - under the hood

Thursday, 30 November 2023

This post is WIP, I might or might go back to this rabbit hole and finish it.

Findings:

1. SemanticsNode already has a Key in it!
   1. Could it just be uploaded to the engine?
   2. Nah, because this key is associated with the Semantics itself???
2. PRs that added support for tooltip in semantics. Something similar could be done with identifier.
   1. framework part https://github.com/flutter/flutter/pull/87684
   2. engine part
      1. original https://github.com/flutter/engine/pull/27893
   3. relanded https://github.com/flutter/engine/pull/28211
3. PR that optimized semantics. Has some stats about how many “semantics” there are in big apps.

How does semantics flow from the Framework to the Engine?

In the framework

1. Developer creates a Semantics widget, passes SemanticsProperties to it
2. The Semantics widget creates its render object – RenderSemanticsAnnotations
3. The RenderSemanticsAnnotations render object calls markNeedsSemanticsUpdate(). This method is inherited from the RenderObject abstract class
4. markNeedsSemanticsUpdate() adds the render object to PipelineOwner._nodesNeedingSemantics.

Also, every RenderObject manages its own SemanticConfiguration.

5. when somebody wants the RenderObject to tell them about its semantics, it calls describeSemanticsConfiguration and passes (TODO)

At some later point, a new frame is scheduled by the engine. This is handled by RendererBinding in the persistent frame callback that it registered. This callback calls RendererBinding’s drawFrame(), which drives the frame rendering pipeline. In one of the last stages of this pipeline, it callsPipelineOwner.flushSemantics(). This gathers _nodesNeedingSemantics.

Finally, SemanticsOwner.sendSemanticsUpdate() is called. This function does (quite some stuff), but most importantly, it finally calls SemanticsUpdateBuilder.build() to obtain a SemanticsUpdate.

💡 SemanticsUpdateBuilder is an abstract class from the engine. It’s implemented in C++ (header) (source).

Then, the SemanticsOwner.onSemanticsUpdate callback is called and the SemanticsUpdate is passed to it as an argument.

(omitted some calls)

Then that SemanticsUpdate is passed to the engine by calling RenderView.updateSemantics().

In the engine

TODO, but generally:

• first, the SemanticsUpdate lands in Engine’s C++ code, and then
• it is delivered to the appropriate embedder APIs
• on Android, JNI is heavily used

Naming observations

• Methods that start with “handle”, e.g. handlePlatformBrightnessChanged, are callbacks to be run when the engine says so

What happens when accessibility is enabled?

“how Flutter learns that it has to start sending accessibility info”

iOS

1. FlutterViewController:

2. PlatformView.h | setSemanticsEnabled

3. PlatformView.cc | setSemanticsEnabled

4. Shell.h | OnPlatformViewSetSemanticsEnabled

5. Shell.cc | OnPlatformViewSetSemanticsEnabled

6. Engine.h | SetSemanticsEnabled

7. Engine.cc | SetSemanticsEnabled

8. RuntimeController.h | SetSemanticsEnabled

9. RuntimeController.cc | SetSemanticsEnabled

10. PlatformConfiguration.h | UpdateSemanticsEnabled

11. PlatformConfiguration.cc | UpdateSemanticsEnabled (crossing C++/Dart boundary)

12. hooks.dart | _updateSemanticsEnabled

13. PlatformDispatcher._updateSemanticsEnabled

14. This callbacks gets invoked: PlatformDispatcher.onSemanticsEnabledChanged

15. That callback is set up by SemanticsBinding: link

The end :)

2024-06-25 – I was awarded Google Open Source Peer Bonus
2024-06-04 – My journey to Google I/O ’24
2024-05-11 – GitHub Actions beg for a supply chain attack
2024-03-19 – Writing a custom Dart VM service extension (part 1)
2024-02-08 – On using smartphone for things that make sense
2023-11-30 – Semantics in Flutter - under the hood
2023-11-25 – Flutter Engine notes
2023-09-17 – Creating and managing Android Virtual Devices using the terminal
2023-05-27 – Suckless Android SDK setup
2023-05-26 – Let’s start over
2023-05-21 – Short thought on “The Zen of Unix”
2023-05-15 – Notes about “flutter assemble”
2019-01-07 – Google Code-in 2018