ls command test development

Considered testing against the GNU specification of the well-known UNIX/Linux command "ls" (List Directory). It includes test architecture design, detailed test design, test case creation, and automated test execution through a CI environment. Note that CI is Failing because some options do not meet the GNU specification, as described below.

SUT

ls command for ubuntsu-latest Ubuntu 20.04.3 LTS as of 2020-11

Test base (Gnu specification)

• Detail page https://www.gnu.org/software/coreutils/manual/html_node/ls-invocation.html#ls-invocation

SUT Source code(Coreutils)

• https://github.com/coreutils/coreutils/blob/master/src/ls.c
• argument switch
  • https://github.com/coreutils/coreutils/blob/5e36c0ce078a65c7dac6ac5ebdfb0cf096856427/src/ls.c#L1916

Test Development/Design Document:

https://github.com/snsk/ls_testing/wiki/test_development

How to test exection

Functional_SpecConfirmance_BasicBehavior

Functional_SpecConfirmance_Arguments_Single_WhichFilesAreListed

Robustness_Consider_Noise

Test Case Execution:

git clone https://github.com/snsk/ls_testing.git
sudo apt install python3-behave
cd ls_testing
behave

*Currently IDs 6, 7, 8, 10, 12, and 15 for "Functional_SpecConfirmance_Arguments_Single_WhichFilesAreListed" fail. This does not appear to be working properly for the GNU specification.

Functional_SpecConfirmance_Arguments_Combination_WhichFilesAreListed

Test Case Execution:

git clone https://github.com/snsk/ls_testing.git
pip install pytest
cd ls_testing
pytest

Robustness_Hierarchy_Fuzzing

Test Case Execution:

git clone https://github.com/snsk/ls_testing.git
pip install randomfiletree
cd ls_testing/HierarchyFuzzing
python3 run_hf.py

Robustness_Arguments_Fuzzing

Test Case Execution:

git clone https://github.com/snsk/ls_testing.git
pip install randomfiletree
cd ls_testing/Arguments_fuzzing
python3 run_af.py

CI Integration Sample

CI by Github Actions is supported(see /workflows/ls_test.yml). In the example below, fuzzing is executed in parallel with Functional Testing because it takes time.

License

MIT