Interact Framework¶
Galah Interact is a framework for creating Test Harnesses that grade students’ programming assignments. For general project information, check out the project page on GitHub: https://github.com/galah-group/galah-interact-python
Galah Interact was originally created by Galah Group LLC and is licensed under the Apache license version 2.0. A copy of the license is available in the root directory of the git repo in the file LICENSE, and online at http://www.apache.org/licenses/LICENSE-2.0. Please ensure your use of this library is within your rights per that license.
Other contributers have given their valuable time to this project in order to make it better, and they are listed in the CONTRIBUTERS file in the root directory of the git repo.
interact.core
¶
Functions and classes that are essential when using this library. Is imported
by interact/__init__.py
such that interact.core.x
and interact.x
are equivalent.
-
class
interact.core.
Harness
[source]¶ An omniscient object responsible for driving the behavior of any Test Harness created using Galah Interact. Create a single one of these when you create your Test Harness and call the
start()
method.A typical Test Harness will roughly follow the below format.
import interact harness = interact.Harness() harness.start() # Your code here harness.run_tests() harness.finish(max_score = some_number)
Variables: - sheep_data – The “configuration” values received from outside the harness (either from Galah or command line arguments). See Test Harness Command Line Interface.
- execution_mode – The mode of execution the harness is running in. Is
set by
Harness.start()
and isNone
before it is set. For information on the different modes, check out Test Harness Command Line Interface. - tests – A dictionary mapping test functions to
Harness.Test
objects. This is of typeORDERED_DICT
.
-
class
FailedDependencies
(max_score=10)[source]¶ A special
TestResult
used byHarness.run_tests()
whenever a test couldn’t be run do to one of its dependencies failing.>>> a = interact.Harness.FailedDependencies() >>> a.add_failure("Program compiles") >>> a.add_failure("Program is sane") >>> print a Score: 0 out of 10 This test will only be run if all of the other tests it depends on pass first. Fix those tests *before* worrying about this one. * Dependency *Program compiles* failed. * Dependency *Program is sane* failed.
-
Harness.
finish
(score=None, max_score=None)[source]¶ Marks the end of the test harness. When start was not initialized via command line arguments, this command will print out the test results in a human readable fashion. Otherwise it will print out JSON appropriate for Galah to read.
-
Harness.
run_tests
()[source]¶ Runs all of the tests the user has registered.
Raises: Harness.CyclicDependency
if a cyclic dependency exists among the test functions.Any tests that can’t be run due to failed dependencies will have instances of
Harness.FailedDependencies
as their result.
-
Harness.
start
(self, arguments = sys.argv[1:])[source]¶ Takes in input from the proper source, initializes the harness with values it needs to function correctly.
Parameters: arguments – A list of command line arguments that will be read to determine the harness’s behavior. See below for more information on this. Returns: None
See also
-
Harness.
student_file
(filename)[source]¶ Given a path to a student’s file relative to the root of the student’s submission, returns an absolute path to that file.
-
interact.core.
ORDERED_DICT
¶ An OrderedDict type. The stdlib’s
collections
module is searched first, then the module ordereddict is searched, and finally it defaults to a regulardict
(which means that the order that test results are displayed will be undefined). This is the type ofHarness.tests
. This complexity is required to support older versions of Python.alias of
OrderedDict
-
class
interact.core.
TestResult
(brief=None, score=None, max_score=None, messages=None, default_message=None, bulleted_messages=True)[source]¶ Represents the result of one unit of testing. The goal is to generate a number of these and then pass them all out of the test harness with a final score.
Variables: - brief – A brief description of the test that was run. This will always be displayed to the user.
- score – The score the student received from this test.
- max_score – The maximum score the student could have received from this test.
- messages – A list of
TestResult.Message
objects that will be joined together appropriately and displayed to the user. Useadd_message()
to add to this list. - default_message – A string that will be displayed if there are no messages. Useful to easily create a “good job” message that is shown when no problems were detected.
- bulleted_messages – A boolean. If
True
, all of the messages will be printed out in bullet point form (a message per bullet). IfFalse
, they will simply be printed out one-per-line. You can set this toFalse
if only one message will ever be displayed to the user, otherwise bullet points usually look better.
-
class
Message
(text, *args, **kwargs)[source]¶ A message to the user. This is the primary mode of giving feedback to users.
-
TestResult.
add_message
(*args, **kwargs)[source]¶ Adds a message object to the TestResult. If a Message object that is used, otherwise a new Message object is constructed and its constructor is passed all the arguments.
-
TestResult.
calculate_score
(starting_score=None, max_score=None, min_score=None)[source]¶ Automatically calculates the score by adding up the
dscore
of each message and setting the score of theTestResult
appropriately.Parameters: - starting_score – This score is added to the sum of every message’s
dscore
. IfNone
,max_score
is used. - max_score – The
max_score
field of the object is set to this value. IfNone
, the currentmax_score
is used, i.e. no change is made. - min_score – If the calculated score is less than this value, this value is used instead.
Returns: self
. This allows you to return the result of this function from test functions.>>> a = TestResult(max_score = 4) >>> a.add_message("Foo", dscore = -1) >>> a.add_message("Bar!", dscore = -5) >>> print a.calculate_score().score -2 >>> print a.score -2 >>> print a.calculate_score(min_score = 0).score 0 >>> print a.calculate_score(starting_score = 8, max_score = 6).score 2 >>> print a.max_score 6
- starting_score – This score is added to the sum of every message’s
-
TestResult.
is_failing
()[source]¶ Returns: The inverse of what is_passing()
returns.This function is most useful when dealing with a
TestResult
that you want to consider either passing or failing, and never anything in between.See also
-
TestResult.
is_passing
()[source]¶ Returns: True
if the score is not0
(note this function will returnTrue
if the score is negative).This function is most useful when dealing with a
TestResult
that you want to consider either passing or failing, and never anything in between.See also
-
TestResult.
set_passing
(passing)[source]¶ Parameters: passing – Whether the test is passing or not. Returns: self
. This allows you to return the result of this function directly, leading to more concise test functions.This function sets
score
to either 1 (ifpassing
isTrue
) or 0 (ifpassing
isFalse
). It also sets themax_score
to1
.See also
is_passing()
andis_failing()
.
-
class
interact.core.
UniverseSet
(iterable=None)[source]¶ A special
set
such that everyin
query returnsTrue
.>>> a = UniverseSet() >>> "hamster" in a True >>> "apple sauce" in a True >>> 3234 in a True >>> "taco" not in a False
-
interact.core.
json_module
()[source]¶ A handy function that will try to find a suitable JSON module to import and return that module (already loaded).
Basically, it tries to load the
json
module, and if that doesn’t exist it tries to load thesimplejson
module. If that doesn’t exist, a friendlyImportError
is raised.
interact.execute
¶
-
interact.execute.
compile_program
(files, flags=[], ignore_cache=False)[source]¶ Compiles the provided code files. If ignore_cache is False and the program has already been compiled with this function, it will not be compiled again.
Parameters: - files – A list of files to compile.
- flags – A list of flags to pass to
g++
. Seecreate_compile_command()
for information on how exactly these are used. - ignore_cache – If
True
, the cache will not be used to service this query, even if an already compiled executable exists. See below for more information on the cache.
Returns: A two-tuple
(compiler output, executable path)
. If the executable was loaded from the cache, the compiler output will beNone
. If the program did not compile successfully, the executable path will beNone
.Note
Note that this function blocks for as long as it takes to compile the files (which might be quite some time). Of coures if the executable is loaded from the cache no such long wait time will occur.
This function caches its results so that if you give it the same files to compile again it will not compile them over again, but rather it will immediately return a prepared executable. The cache is cleared whenever the program exits.
-
interact.execute.
create_compile_command
(files, flags)[source]¶ From a list of files and flags, crafts a list suitable to pass into
subprocess.Popen
to compile those files.Parameters: - files – A list of files to compile.
- flags – A list of flags to pass onto
g++
.-o main
will always be passed after these flags.
Returns: A list of arguments appropriate to pass onto
subprocess.Popen
.>>> create_compile_command(["main.cpp", "foo.cpp"], ["-Wall", "-Werror]) ["g++", "-Wall", "-Werror", "-o", "main", "main.cpp", "foo.cpp"]
-
interact.execute.
default_run_func
(executable, temp_dir, args=[])[source]¶ Used by the
run_program()
to create aPopen
object that is responsible for running the exectuable.Parameters: - executable – An absolute path to the executable that needs to be run.
- temp_dir – An absolute path to a temporary directory that can be
used as the current working directory. It will be deleted
automatically at the end of the
run_program()
function. The executable will not be in the directory. - args – A list of arguments to give the executabe.
This function may be overriden to override the default
run_func
value used in therun_program()
function.Warning
You must pass in
subprocess.PIPE
to thePopen
constructor for thestdout
andstdin
arguments.You can use this function as a reference when creating your own run functions to pass into
run_program()
.
-
interact.execute.
run_program
(files=None, given_input='', run_func=None, executable=None, timeout=None, args=[])[source]¶ Runs a program made up of some code files by first compiling, then executing it.
Parameters: - files – The code files to compile and execute.
compile_program()
is used to compile the files, so its caching applies here. - given_input – Text to feed into the compiled program’s standard input.
- run_func – A function responsible for creating the
Popen
object that actually runs the program. Defaults todefault_run_func()
. - executable – If you don’t need to compile any code you can pass a path to an executable that will be executed directly.
- timeout – Specifies, in seconds, when process should be terminated.
returncode
will be None if terminated forcefully. - args – Gives arguments to the executable.
Returns: A three-tuple containing the result of the program’s execution
(stdout, stderr, returncode)
.- files – The code files to compile and execute.
interact.parse
¶
This module is useful when attempting to roughly parse students’ code (ex: trying to check that indentation was properly used). This module does not attempt to, and never will, try and fully parse C++. If such facilities are added to Galah Interact they will probably be added as a seperate module that provides a nice abstraction to Clang.
-
class
interact.parse.
Block
(lines, sub_blocks=None)[source]¶ Represents a block of code.
Variables: - lines – A list of
Line
objects that make up this block. - sub_blocks – A list of
Block
objects that are children of this block.
- lines – A list of
-
interact.parse.
INDENT_EXCEPTED_LINES
= ['public:', 'private:', 'protected:']¶ Lines of code to ignore when looking for bad indentation. See
find_bad_indentation()
for more information.
-
class
interact.parse.
Line
(line_number, code)[source]¶ Represents a line of code.
Variables: - code – The contents of the line.
- line_number – The line number.
-
indent_level
()[source]¶ Determines the indentation level of the current line.
Returns: The sum of the number of tabs and the number of spaces at the start of the line. Iff the line is blank (not including whitespace), None
is returned.
-
static
lines_to_str
(lines)[source]¶ Creates a single string from a list of
Line
objects.Parameters: lines – A list of Line
objects.Returns: A single string. >>> my_lines = [ Line(1, "int main() {"), Line(2, " return 0;"), Line(3, "}") ] >>> Line.lines_to_str(my_lines) "int main() {\n return 0\n}\n"
-
static
lines_to_str_list
(lines)[source]¶ Creates a list of strings from a list of
Line
objects.Parameters: lines – A list of Line
objects.Returns: A list of strings. >>> my_lines = [ Line(1, "int main() {"), Line(2, " return 0;"), Line(3, "}") ] >>> Line.lines_to_str_list(my_lines) [ "int main() {", " return 0;", "}" ]
-
classmethod
make_lines
(lines, start=1)[source]¶ Creates a list of Line objects from a list of strings representing lines in a file.
Parameters: - lines – A list of strings where each string is a line in a file.
- start – The line number of the first line in
lines
.
Returns: A list of line objects.
>>> Line.make_lines(["int main() {", " return 0;", "}"], 1) [ Line(1, "int main() {"), Line(2, " return 0;"), Line(3, "}") ]
-
interact.parse.
cleanse_quoted_strings
(line)[source]¶ Removes all quoted strings from a line. Single quotes are treated the same as double quotes.
Escaped quotes are handled. A forward slash is assumed to be the escape character. Escape sequences are not processed (meaning “ does not become “, it just remains as “).
Parameters: line – A string to be cleansed. Returns: The line without any quoted strings. >>> cleanse_quoted_strings("I am 'John Sullivan', creator of worlds.") "I am , creator of worlds." >>> cleanse_quoted_strings( ... 'I am "John Sullivan \"the Destroyer\", McGee", fear me.' ... ) "I am , fear me."
This function is of particular use when trying to detect curly braces or other language constructs, and you don’t want to be fooled by the symbols appearing in string literals.
-
interact.parse.
find_bad_indentation
(block, minimum=None)[source]¶ Detects blocks of code that are not indented more than their parent blocks.
Parameters: - block – The top-level block of code. Sub-blocks will be recursively checked.
- minimum – The minimum level of indentation required for the top-level block. Mainly useful due to this function’s recursive nature.
Returns: A list of
Line
objects where eachLine
had a problem with its indentation.Note
Lines that match (after removing whitespace) lines in
INDENT_EXCEPTED_LINES
will be ignored.>>> my_block = Block( ... lines = [ ... Line(0, "#include <iostream>"), ... Line(1, ""), ... Line(2, "using namespace std;"), ... Line(3, ""), ... Line(4, "int main() {"), ... Line(15, "}") ... ], ... sub_blocks = [ ... Block( ... lines = [ ... Line(5, ' cout << "{" << endl;'), ... Line(6, " if (true)"), ... Line(7, " {"), ... Line(9, " } else {"), ... Line(12, " }"), ... Line(13, " pinata"), ... Line(14, " return 0") ... ], ... sub_blocks = [ ... Block( ... lines = [ ... Line(8, " return false;") ... ] ... ), ... Block( ... lines = [ ... Line(10, " return true;"), ... Line(11, "oh noz") ... ] ... ) ... ] ... ) ... ] ... ) >>> find_bad_indentation(my_block) [Line(11, "oh noz")]
-
interact.parse.
grab_blocks
(lines)[source]¶ Finds all blocks created using curly braces (does not handle two line if statements for example).
Parameters: lines – A list of Line
objects.Returns: A single Block
object which can be traversed like a tree.>>> my_lines = [ ... Line(0, "#include <iostream>"), ... Line(1, ""), ... Line(2, "using namespace std;"), ... Line(3, ""), ... Line(4, "int main() {"), ... Line(5, ' cout << "Hello world" << endl;'), ... Line(6, " return 0"), ... Line(7, "}") ... ] >>> grab_blocks(my_lines) Block( lines = [ Line(0, "#include <iostream>"), Line(1, ""), Line(2, "using namespace std;"), Line(3, ""), Line(4, "int main() {"), Line(7, "}") ], sub_blocks = [ Block( lines = [ Line(5, ' cout << "Hello world" << endl;'), Line(6, " return 0") ], sub_blocks = None ) ] )
(Note that I formatted the above example specially, it won’t actually print out so beautifully if you try it yourself, but the content will be the same)
interact.pretty
¶
Module useful for displaying nice, grammatically correct output.
-
interact.pretty.
craft_shell_command
(command)[source]¶ Returns a shell command from a list of arguments suitable to be passed into subprocess.Popen. The returned string should only be used for display purposes and is not secure enough to actually be sent into a shell.
-
interact.pretty.
escape_shell_string
(str)[source]¶ Escapes a shell string such that it is suitable to be displayed to the user. This function should not be used to actually feed arguments into a shell as this function is not secure enough.
-
interact.pretty.
plural_if
(zstring, zcondition)[source]¶ Returns zstring pluralized (adds an ‘s’ to the end) if zcondition is True or if zcondition is not equal to 1.
Example usage could be
plural_if("cow", len(cow_list))
.
-
interact.pretty.
pretty_list
(the_list, conjunction='and', none_string='nothing')[source]¶ Returns a grammatically correct string representing the given list. For example...
>>> pretty_list(["John", "Bill", "Stacy"]) "John, Bill, and Stacy" >>> pretty_list(["Bill", "Jorgan"], "or") "Bill or Jorgan" >>> pretty_list([], none_string = "nobody") "nobody"
interact.standardtests
¶
This module contains useful test functions that perform full testing on input, returning TestResult objects. These are typical tests that many harnesses need to perform such as checking indentation or checking to see if the correct files were submitted.
-
interact.standardtests.
check_compiles
(files, flags=[], ignore_cache=False)[source]¶ Attempts to compile some files.
Parameters: - files – A list of paths to files to compile.
- flags – A list of command line arguments to supply to the compiler.
Note that
-o main
will be added after your arguments. - ignore_cache – If you ask Galah Interact to compile some files, it
will cache the results. The next time you try to compile the same
files, the executable that was cached will be used instead. Set
this argument to
True
if you don’t want the cache to be used.
Returns: A
TestResult
object.>>> print interact.standardtests.check_compiles(["main.cpp", "foo.cpp"]) Score: 0 out of 10 This test ensures that your code compiles without errors. Your program was compiled with g++ -o main /tmp/main.cpp /tmp/foo.cpp. Your code did not compile. The compiler outputted the following errors: ``` /tmp/main.cpp: In function 'int main()': /tmp/main.cpp:7:9: error: 'foo' was not declared in this scope /tmp/main.cpp:9:18: error: 'dothings' was not declared in this scope /tmp/main.cpp:11:19: error: 'dootherthings' was not declared in this scope ```
-
interact.standardtests.
check_files_exist
(*files, **extra)[source]¶ Checks to see if the given files provided as arguments exist. They must be files as defined by os.path.isfile().
Parameters: - *files – The files to check for existance. Note this is not a list, rather you should pass in each file as a seperate arugment. See the examples below.
- **extra – extra parameters. If extra[“basename”] is True, then os.path.basename is applied to all filenames before printing.
Returns: Returns a TestResult object that will be passing iff all of the files exist.
# The current directory contains only a main.cpp file. >>> print check_files_exist("main.cpp", "foo.cpp", "foo.h") Score: 0 out of 1 This test ensures that all of the necessary files are present. * You are missing foo.cpp and foo.h.
(Note that this function really does return a TestResult object, but TestResult.__str__() which transforms the TestResult into a string that can be printed formats it specially as seen above)
-
interact.standardtests.
check_indentation
(files, max_score=10, allow_negative=False)[source]¶ Checks to see if code is indented properly.
Currently code is indented properly iff every block of code is indented strictly more than its parent block.
Parameters: - files – A list of file paths that will each be opened and examined.
- max_score – For every improperly indented line of code, a point is
taken off from the total score. The total score starts at
max_score
. - allow_negative – If True, a negative total score will be possible, if False, 0 will be the lowest score possible.
Returns: A
TestResult
object.>>> print open("main.cpp").read() #include <iostream> using namespace std; int main() { if (true) { foo(); } else { dothings(); while (false) { dootherthings(); } cout << "{}{}{{{{{}}}{}{}{}}}}}}}}{{{{"<< endl; } return 0; } >>> print open("foo.cpp").read() #include <iostream> using namespace std; int main() { return 0; } >>> print check_indentation(["main.cpp", "foo.cpp"]) Score: 6 out of 10 This test checks to ensure you are indenting properly. Make sure that every time you start a new block (curly braces delimit blocks) you indent more. * Lines 14, 13, and 10 in main.cpp are not indented more than the outer block. * Line 5 in foo.cpp is not indented more than the outer block.
interact.unittest
¶
This module contains very useful functions you can use while unittesting student’s code.
Note
In order to use the unittest
module, you need to make sure that you
have SWIG installed, and that you have Python development headers
installed, both of which are probably available through your distribution’s
package manager (apt-get
or yum
for example).
-
exception
interact.unittest.
CouldNotCompile
(message, stderr)[source]¶ Exception raised when a student’s code could not be compiled into a single library file.
Variables: - message – A short message describing the exception.
- stderr – The output that was received through standard error. This is
output by
distutils.core.setup
.
-
interact.unittest.
load_files
(files)[source]¶ Compiles and loads functions and classes in code files and makes them callable from within Python.
Parameters: files – A list of file paths. All of the files will be compiled and loaded together. These must be absolute paths, see Harness.student_files
.Returns: A dict
where every file that was passed in is a key in the dictionary (without its file extension) and the value is anotherdict
where each key is the name of a function or class in the file and the value is a callable you can use to actually execute or create an instance of that function or class.Raises: EnvironmentError
if swig is not properly installed.Raises: CouldNotCompile
if the student’s code could not be compiled into a library file.Warning
During testing, oftentimes the execution of loaded code’s
main()
function failed. We haven’t determined what the problem is yet so for now don’t use this function to testmain()
functions (theinteract.execute
module should work well instead).>>> print open("main.cpp").read() #include <iostream> using namespace std; class Foo { int a_; public: Foo(int a); int get_a() const; }; Foo::Foo(int a) : a_(a) { // Do nothing } int Foo::get_a() const { return a_; } int bar() { Foo foo(3); cout << "foo.get_a() = " << foo.get_a() << endl; return 2; } int main() { return 0; } >>> students_code = interact.unittest.load_files(["main.cpp"]) >>> Foo = students_code["main"]["Foo"] >>> bar = students_code["main"]["bar"] >>> b = Foo(3) >>> b.get_a() 3 >>> rvalue = b.bar() foo.get_a() = 3 >>> print rvalue 2
If you want to test a function that prints things to stdout or reads from stdin (like the
bar()
function in the above example) you can use theinteract.capture
module.
-
interact.unittest.
swig_path
= None¶ The absolute path to the swig executable. When this module is imported, the environmental variable
PATH
is searched for a file namedswig
, this variable will be set to the first one that is found. This variable will equalNone
if no such file could be found.
interact.capture
¶
This module provides the tools for easily running a Python function in a seperate process in order to capture its standard input, output, and error.
-
class
interact.capture.
CapturedFunction
(pid, stdin_pipe, stdout_pipe, stderr_pipe, returnvalue_pipe)[source]¶ The type of object returned by
capture_function()
. Provides access to a captured function’s stdin, stdout, stderr, and return value.Variables: - pid – The process ID of the process that is running/ran the function.
- stdin – A file object (opened for writing) that the captured function is using as stdin.
- stdout – A file object (opened for reading) that the captured function is using as stdout.
- stderr – A file object (opened for reading) that the captured function is using as stderr.
- return_value – Whatever the function returned. Will not be set until
CapturedFunction.wait()
is called. Will contain the valueCapturedFunction.NOT_SET
if it has not been set by a call toCapturedFunction.wait()
.
The correct way to check if
return_value
is set is to compare withCapturedFunction.NOT_SET
like so:if my_captured_function.return_value is CapturedFunction.NOT_SET: print "Not set yet!" else: print "It's set!"
-
NOT_SET
= <interact.capture._NotSet instance>¶ A sentinel value used to denote that a
return_value
has not been set yet.
-
interact.capture.
capture_function
(func, *args, **kwargs)[source]¶ Executes a function and captures anything it prints to standard output or standard error, along with capturing its return value.
Parameters: - func – The function to execute and capture.
- *args,**kwargs – The arguments to pass to the function.
Returns: An instance of
CapturedFunction
.>>> def foo(x, c = 3): ... print x, "likes", c ... return x + c >>> a = capture_function(foo, 2, c = 9) >>> a.stdout.read() "2 likes 9\n" >>> a.wait() >>> print a.return_value 11