Add a script to build TF-A and measure the code size - #219
Open
davidhorstmann-arm wants to merge 33 commits into
Open
Add a script to build TF-A and measure the code size#219davidhorstmann-arm wants to merge 33 commits into
davidhorstmann-arm wants to merge 33 commits into
Conversation
Add a script to get the code size of TF-A, given a path to a TF-A tree. Currently we use hard-coded build options but future changes will add the capability to read build configs from TF-A's CI Signed-off-by: David Horstmann <david.horstmann@arm.com>
Require 2 additional arguments: * A TF-A build config name such as fvp-tbb-mbedtls-ecdsa * The path to the tf-a-ci-scripts repository[1], which maps these config names to sets of options for make. Use these to perform the mapping and call make with the specified build config. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Signed-off-by: David Horstmann <david.horstmann@arm.com>
Add an argument to pass the Mbed TLS directory. Since we are measuring how code size changes with different Mbed TLS versions, it is important that we can pass a custom Mbed TLS directory. Thankfully TF-A has an option for us to do just that. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Subject the code size measurement script to the 90s, while keeping the functionality the same. We now have a class called Configuration which represents a combination of TF-A version, TF-A config and Mbed TLS version. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Put the commit hashes and the config in the filename. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Allow specifying the commits used in TF-A and Mbed TLS. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Before measuring, make sure the correct commits of TF-A and Mbed TLS are checked out. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Standardize the ref type by using git rev-parse to get a short ref. This allows us to have different lengths of hash and even branches or tags and normalise them into a short ref before using them as the unique size report name. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Summing the symbols from nm should give us a good enough approximation to size without having everything rounded up to the next page size. The types map to (approximately) .text, .rodata, .data and .bss but there is some possibility of other things appearing and of aliasing. It's probably good enough for what we need. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Some of the symbols mark the start or end of a section, therefore have big sizes, possibly reflecting the size of the section. However, they do not reflect real code symbols that consume space. Filter out these symbols so we no longer have them in the output. Signed-off-by: David Horstmann <david.horstmann@arm.com>
This should make tab-completion easier when we need to delete a report. Signed-off-by: David Horstmann <david.horstmann@arm.com>
If they are not supplied we will clone them to a working directory (or not do that if they are already cloned). Signed-off-by: David Horstmann <david.horstmann@arm.com>
We may have submodules changing between revisions, so re-update them. Signed-off-by: David Horstmann <david.horstmann@arm.com>
When we have a branch name it will not reference anything locally but will be a valid name in origin. For convenience, if we can't find the ref in the local checkout, try prepending 'origin/'. This also requires us to fetch everything when we ensure the repo is cloned, to make sure we have fresh information. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Instead of one revision of TF-A and Mbed TLS and a single build config, take a list of config arguments using a simple syntax. This will make it more convenient to build multiple configs in one go for testing. In future we will add a part to compare all of the configs, but for now we can just build them all and generate the code size data. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Since we name by ref we can check if there is already a report for that ref combination, saving time. Signed-off-by: David Horstmann <david.horstmann@arm.com>
This avoids us cluttering up the place where the script is run. Signed-off-by: David Horstmann <david.horstmann@arm.com>
When comparing multiple versions, we would like to find: * All of the symbols that are in one of the compared configs but not in the base config and vice versa. * All of the symbols that differ in size between the base config and at least one other config. Create a function to perform this computation for us. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Add a method for the configuration which generates the report if necessary, reads the file and returns the JSON. Signed-off-by: David Horstmann <david.horstmann@arm.com>
The regex was not catching 'RAM_REGION_START' due to the leading double-underscore requirement, causing the size measurements to be negative somehow. Signed-off-by: David Horstmann <david.horstmann@arm.com>
For now, just get the total sizes of each config and lay them out in a row. In future we will add: * Overall deltas of total size. * Symbol-by-symbol deltas for interesting symbols (those that changed). Signed-off-by: David Horstmann <david.horstmann@arm.com>
Generate a prettily formatted table listing the total sizes for all configurations, their differences in bytes from the base configuration and a list of the symbols that differ, along with how many bytes they differ by. Limitations: * Currently does not generate titles for the configs properly. In future we will add fancy logic to do this nicely. * Currently only generates the table in an array. We will print it properly in a future change. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Previously we were normalising the names to git short refs. Still do this for internal operations but keep a reference to the user-specified name as well so that we can use it in pretty-printing. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Deal with the case where a symbol exists in one config but not in another when generating the tables. We are fortunate here that Python's format() function is happy for us to pass '-' to something that would like to format a number and we still seem to get the correct spacing. Signed-off-by: David Horstmann <david.horstmann@arm.com>
So as not to be overly verbose, isolate the elements of configurations that differ from each other and print only those elements. For example if we were only changing the Mbed TLS commits, we could title each column based on its Mbed TLS commit. Signed-off-by: David Horstmann <david.horstmann@arm.com>
When we have a symbol that does exist in one config but not in base or the config currently being analysed, put a 0 in the relevant row. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Use the existing name_diff method to figure out the differing elements of the config and use them as table titles. Auto-size the columns of the table so that they line up nicely. Note that if more then one element differs, the table is likely to be too wide to fit on a screen. In this case we should really do some clever multiline formatting but that can come later if we need it. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Instead of doing formatting things in the gen_comparison_table() function, get that function to create a Table object, which is a wrapper around a list of titles and a list of rows of entries, each of which is either a single value or a tuple of (size, diff). Move the formatting code to a method of the Table class. This will simplify adding a markdown output mode for convenience. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Add a method to the Table class which generates a table in GitHub-flavour markdown. This will be convenient for displaying the results in issues on GitHub and other forums that support this format. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Allow selecting markdown tables via a command-line switch (the default is CLI-formatted tables, which have spacing). Note that CLI-formatting is not very good if the configs differ in many aspects (so the titles have very long names) as the formatted rows do not fit on single monitor lines. Signed-off-by: David Horstmann <david.horstmann@arm.com>
Previously we were putting [Equal] as the title because there was no distinguisher between the different configs. This was ugly and unhelpful so change to just using the TF-A name by default (after all, that's the thing we're measuring the size of in the end). Signed-off-by: David Horstmann <david.horstmann@arm.com>
Explain more specifically what the script does and explain what a 'config' means in this rather specific context. Signed-off-by: David Horstmann <david.horstmann@arm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This script downloads and builds TF-A with the specified configs and generates a table comparing the code size, either formatted for the CLI or in GitHub-flavour markdown.
Running it requires the TF-A build prerequisites to be installed.
TF-A's
integrationbranch seems happy to build againstmbedtls-3.6.6andmbedtls-3.6.7but builds against other versions fail. The script may be tested out with:Which builds the integration branch against those 2 release tags of Mbed TLS in the
fvp-tbb-mbedtls-ecdsaconfig.This should generate an output similar to: