Statement testing with MS Copilot

Task 1 (student 1)

Using MS Copilot identify statements for the insertSort function. Assign a tag to for each identified statement. Present the list of statements with tags. Write the prompt and the chatbot answer. Discuss the result.

Does it properly recognized that the FOR instruction consists of 3 statements?

Task 2 (student 2)

Using MS Copilot generate the Control Flow Graph in the DOT language for the insertSort function. Do not force Copilot to create the instruction list. Write the prompt and the chatbot answer. Discuss the result.

Does the CFG has the same structure as the CFG created by a human in the previous section of the book?

When the Task 1 is finished, use the generated instruction list with tags as the hint for Copilot. Did it help them co correct the CFG (now the CFG should be correct)? Discuss the result.

Prompt: Generate for me Control Flow Graph in the DOT language for the insertSort function from wikipedia.

Answer: Certainly! Below is the Control Flow Graph (CFG) of the insertSort function in the DOT language:

digraph InsertSortCFG {
entry [label="Entry"];
exit [label="Exit"];

entry -> init_i;
init_i -> cond_i [label="i = 1"];
cond_i -> exit [label="i >= length"];
cond_i -> init_value [label="i < length"];

init_value -> init_j [label="value = a[i]"];
init_j -> cond_j [label="j = i - 1"];
cond_j -> assign_a_jp1 [label="a[j + 1] = value"];
cond_j -> body_j [label="j >= 0 && a[j] > value"];

body_j -> shift [label="a[j + 1] = a[j]"];
shift -> update_j;
update_j -> cond_j [label="j--"];

assign_a_jp1 -> update_i;
update_i -> cond_i [label="i++"];
}

Comparing the graph to the one created by human, we can see that the chatbot didn't succeed. The structure of the program is more or less correct (the "Shift" node is an anomaly), however the graph itself is not a proper Control Flow Graph.

Task 3. (student 3)

Prompt:

write an instruction test for the function insertSort in the C language. Here is the function:
void insertSort(int a[], int length)
{
int i, j, value;
for (i = 1; i < length; ++i)
{
value = a[i];
for (j = i - 1; j >= 0 && a[j] > value; --j)
{
a[j + 1] = a[j];
}
a[j + 1] = value;
}
}
Keep in mind to precisely test various data types and values, compare them with expected results

Answer:

Sure! Here's an instruction test for the insertSort function in C. This test will cover various data types and values, and compare them with the expected results.

#include <stdio.h>

#include <assert.h>

// Function prototype

void insertSort(int a[], int length);

// Function to print array (for debugging purposes)

void printArray(int a[], int length) {

for (int i = 0; i < length; i++) {

printf("%d ", a[i]);

}

printf("\n");

}

// Test function

void testInsertSort() {

// Test case 1: Normal case

int arr1[] = {5, 2, 9, 1, 5, 6};

int expected1[] = {1, 2, 5, 5, 6, 9};

insertSort(arr1, 6);

for (int i = 0; i < 6; i++) {

assert(arr1[i] == expected1[i]);

}

// Test case 2: Already sorted array

int arr2[] = {1, 2, 3, 4, 5};

int expected2[] = {1, 2, 3, 4, 5};

insertSort(arr2, 5);

for (int i = 0; i < 5; i++) {

assert(arr2[i] == expected2[i]);

}

// Test case 3: Array with negative numbers

int arr3[] = {-3, -1, -2, -5, -4};

int expected3[] = {-5, -4, -3, -2, -1};

insertSort(arr3, 5);

for (int i = 0; i < 5; i++) {

assert(arr3[i] == expected3[i]);

}

// Test case 4: Array with duplicates

int arr4[] = {4, 2, 2, 8, 3, 3, 1};

int expected4[] = {1, 2, 2, 3, 3, 4, 8};

insertSort(arr4, 7);

for (int i = 0; i < 7; i++) {

assert(arr4[i] == expected4[i]);

}

// Test case 5: Single element array

int arr5[] = {1};

int expected5[] = {1};

insertSort(arr5, 1);

assert(arr5[0] == expected5[0]);

// Test case 6: Empty array

int arr6[] = {};

int expected6[] = {};

insertSort(arr6, 0);

// No assertion needed for empty array

printf("All test cases passed!\n");

}

int main() {

testInsertSort();

return 0;

}

This test function covers various scenarios:

A normal case with mixed values.
An already sorted array.
An array with negative numbers.
An array with duplicate values.
A single element array.
An empty array.

Each test case uses assert to compare the sorted array with the expected result. If all assertions pass, it prints "All test cases passed!" to indicate success.

Summary

Copilot created seperate function for printing array elements, which is good as it provides good code organization. Whole testing takes place in function testInsertSort(). There are 6 cases, including common numbers array, negative numbers array, empty array, sorted array, single element list and array with duplicates.

Running the code revealed an important issue with function:

main.c: In function ‘testInsertSort’:
main.c:77:5: warning: ‘insertSort’ accessing 4 bytes in a region of size 0 [-Wstringop-overflow=]
77 | insertSort(arr6, 0);
| ^~~~~~~~~~~~~~~~~~~
main.c:77:5: note: referencing argument 1 of type ‘int[0]’
main.c:11:6: note: in a call to function ‘insertSort’
11 | void insertSort(int a[], int length)
| ^~~~~~~~~~
All test cases passed!

It turns out, that when empty array is passed, there is an overflow, as j value is not initialazed it defaults to 0. Then it tries to reach a[1] element of array, which triggers overflow warning. Even though, the code still runs and returns 0.

Copilot failed to test cases with extremely big numbers.